An intelligent data management system and method based on 5G communication

Abstract

The present invention discloses an intelligent data management system and method based on 5G communication, which belongs to the field of artificial intelligence technology. The present invention collects the operation data of all electronic devices of the company in real time through the 5G network, performs preprocessing, and integrates them into an electronic device data set; based on the similarity of data communication frequency and data content, analyzes the association between the electronic devices of the company; represents the electronic devices and their association relationships as a graph structure, and obtains the association parameters between the devices; arranges the association parameters in descending order according to the breadth and intensity of the association, forms an importance sequence of the electronic devices, and removes the redundant parameters in the electronic devices; according to the importance sequence of the electronic devices, sets the data management priority and security level for each device; uses the isolation forest algorithm to detect abnormalities in the data, and checks whether there are errors in the data of other devices associated with it according to the priority of the importance sequence of the electronic devices; and performs data repair.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to an intelligent data management system and method based on 5G communication. Background Art

[0002] With the continuous development of science and technology and the popularization of 5G communication technology, enterprises are increasingly relying on electronic devices to improve production efficiency. The widespread use of electronic devices has brought a large amount of operating data, including equipment status, performance indicators, and usage. By analyzing and managing this data, the operating efficiency and safety of the equipment can be effectively improved. However, traditional data management methods have many shortcomings and cannot meet the needs of modern enterprises for data efficiency.

[0003] Traditional data collection methods often rely on wired networks or low-speed wireless networks, which have large data transmission delays and cannot achieve real-time data collection. Existing data processing systems usually use a single data processing algorithm with low processing efficiency and are difficult to meet the needs of large-scale data processing. Traditional association analysis methods are simple and cannot accurately capture the complex association relationships between devices, resulting in inaccurate association analysis results. Existing systems lack intelligent data management methods and cannot intelligently set priorities and security levels based on the importance of devices. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent data management system and method based on 5G communication to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] An intelligent data management method based on 5G communication, the method comprising the following steps:

[0007] S100, collecting operation data of all electronic devices of the company in real time through the 5G network, wherein the operation data includes device status, performance indicators and usage, preprocessing the operation data, and integrating the preprocessed operation data into an electronic device data set;

[0008] S200, according to the electronic device data set, based on the data communication frequency and the similarity of the data content, use the K-means algorithm and the Apriori algorithm to analyze the association between the electronic devices of the company; represent the electronic devices and their association relationships as a graph structure, and obtain the association parameters between the devices;

[0009] S300, according to the correlation parameters, the correlation breadth and strength are arranged in descending order to form an importance sequence of the electronic devices, and the Lasso regression algorithm is used to remove redundant parameters in the electronic devices and retain key parameters; according to the importance sequence of the electronic devices, a data management priority and a security level are set for each device, so that important devices can receive higher priority management and higher level security protection;

[0010] S400. Use the isolation forest algorithm to monitor the data of electronic devices in real time and detect anomalies in the data. When an error occurs in the data of a certain electronic device, check whether the data of other devices associated with it have errors according to the priority sequence of the importance of the electronic device; if the data of the associated device has no errors, overwrite the erroneous data with the correct data; if the data of all associated devices have errors, use the remaining correct data of the electronic device containing the erroneous data to deduce through a convolutional neural network and perform data repair.

[0011] According to step S100, the electronic device integrates a 5G communication module, uses the SSL / TLS encryption protocol to encrypt data before transmission, and uses the MQTT protocol as the data transmission protocol for transmission; uses Alibaba Cloud as the data receiving and storage center, and deploys a load balancer; designs a data receiving API interface to implement data writing operations, and uses Kafka as a message queue to buffer and distribute real-time data; cleans the received data at the data receiving and storage center, removes invalid data and duplicate data, and converts the data into a unified format; stores the preprocessed data in a PostgreSQL database, classifies and labels the data, and integrates the preprocessed operating data into an electronic device data set.

[0012] According to step S200, the data communication frequency between each pair of electronic devices is calculated. The higher the communication frequency, the higher the correlation between the two devices. The data communication frequency is obtained by weighted calculation by counting the number of communications and the amount of communication data between each pair of devices over a period of time; the data content of device status, performance indicators and usage is analyzed to calculate their similarity; the status change patterns of different devices are compared to obtain status similarity, the performance indicators of different devices are compared to obtain performance indicator similarity, and the usage patterns of the devices are compared to obtain usage similarity;

[0013] Use the K-means algorithm to group devices according to their features and obtain device clusters with similar communication frequency and content features. Use the Apriori algorithm to mine the association patterns between devices and find device combinations and association rules. Combine the communication frequency features and content similarity features to construct a comprehensive feature vector, with each device represented by a feature vector.

[0014] When using the K-means algorithm to perform cluster analysis on feature vectors, several cluster centers are randomly initialized, the distance from each device to the cluster center is calculated, the device is assigned to the nearest cluster center, and the position of the cluster center is updated until the clustering results converge; the clustering results reflect the similarity and correlation between devices, and the devices in the same cluster have a high correlation;

[0015] The Apriori algorithm is used to iteratively generate frequent item sets, calculate the frequency of occurrence of each pair of devices, generate association rules, evaluate the support and confidence of the rules, and screen out the association rules with the highest support and confidence.

[0016] According to step S200, each electronic device is represented as a node, and each node contains basic information and features of the device; the edge represents the association relationship between the devices, and the weight of the edge represents the strength of the association between the devices; the direction of the edge allows directed edges and undirected edges, wherein directed edges represent unidirectional associations, and undirected edges represent bidirectional associations; a list containing all electronic device nodes is created, and an empty edge list is initialized to store the association relationships between the devices; the frequency of data communication between each pair of devices is counted, and the higher the frequency, the stronger the association; the data content of the device status, performance indicators and usage is compared, and their similarity is calculated, and the higher the similarity, the stronger the association; based on the calculated association, it is decided whether to add an edge between the nodes and the weight of the edge, and if the association between the two devices exceeds a certain threshold, an edge is added between the two devices and a weight is assigned;

[0017] Calculate the association parameters between devices, use Gephi to visualize the graph structure, and display the electronic devices and their association relationships; regularly recalculate the association parameters between devices so that the graph structure reflects the latest device association status; update the nodes and edges in the graph structure when adding new devices or removing faulty devices.

[0018] The association parameters between the devices include node degree, edge weight, adjacency matrix, node clustering coefficient, shortest path and node centrality; wherein the node degree is the number of edges directly connected to the node, reflecting the number of direct associations of the device. The higher the node degree, the wider the association of the device. The edge weight indicates the strength of the association between the devices. The larger the edge weight, the stronger the association between the devices. The edge weight is calculated comprehensively based on the communication frequency and content similarity. The adjacency matrix is ​​a square matrix. The elements of the matrix represent the association relationship between the nodes. The element value can be 0 or a weight value. 0 indicates no association, and a weight value indicates related The adjacency matrix is ​​used to query the direct association between devices. The clustering coefficient of a node represents the ratio of the number of actual edges between a node and its neighboring nodes to the number of possible edges. The clustering coefficient reflects the density of devices in the local network. The higher the coefficient, the stronger the association of the local network where the device is located. The shortest path is the path with the smallest weight among the paths connecting two nodes. The shortest path is used to measure the indirect association between devices. The shorter the path, the stronger the indirect association between devices. The node centrality is a measure of the importance of the node in the entire network, which is calculated through degree centrality, closeness centrality and betweenness centrality.

[0019] According to step S300, for each electronic device, a comprehensive importance score is calculated according to the breadth and strength of its associated parameters, and the importance of each parameter is weighted and summed to obtain a comprehensive importance score for each device; all electronic devices are arranged in descending order according to the comprehensive importance score to form an importance sequence of the electronic devices;

[0020] The importance score of the equipment is used as the independent variable, and other business-related target variables are used as dependent variables to construct a data set; the data is standardized to ensure that the numerical range of all parameters is consistent, and the data set is divided into a training set and a test set; the Lasso regression algorithm is used for model training, and by adding absolute value regularization terms to the loss function, the coefficients of certain parameters tend to zero, thereby achieving parameter selection and dimensionality reduction; during the training process, Lasso regression will gradually reduce the coefficient values ​​of unimportant parameters to zero to remove redundant parameters; after the training is completed, check the coefficient values ​​of each parameter in the model, and retain the parameters with non-zero coefficient values ​​as the key parameters that have a greater impact on the target variable; the parameters that are reduced to zero are redundant parameters and are removed; use the test set to evaluate the model performance, and adjust and optimize the parameter selection based on the model evaluation results.

[0021] According to step S300, the data management priority is divided into three levels: high, medium and low, wherein the high priority requires the most real-time and most frequent data collection and processing, the medium priority requires regular data collection and processing, and the low priority performs batch data collection and processing; the security level is divided into three levels: high, medium and low, wherein the high security level requires the highest level of security protection, including encrypted transmission, storage encryption, access control, real-time monitoring and anomaly detection, the medium security level requires medium-level security protection, including encrypted transmission, storage encryption, regular security audits and anomaly detection, and the low security level requires basic security protection, including encrypted transmission, access control and anomaly detection;

[0022] Encrypted transmission refers to the use of encryption technology to protect the confidentiality and integrity of data during data transmission. By using TLS transport layer security or SSL secure socket layer protocol to encrypt data, it is ensured that data will not be stolen or tampered with by unauthorized third parties during transmission. Storage encryption refers to the encryption protection of data stored on storage media such as hard disks and databases, and the use of AES encryption algorithm to encrypt and store data to ensure that even if the storage medium is stolen or illegally accessed, the data is still unreadable. Access control refers to controlling user and system access to data and resources through permission management, setting roles and permissions, ensuring that only authorized users and systems can access specific data and resources, and preventing unauthorized access. Real-time monitoring refers to continuous and real-time monitoring of systems and data to promptly discover and respond to abnormal situations, deploy monitoring systems, collect and analyze system logs, performance indicators and other data in real time, and promptly discover and alarm abnormal situations. Anomaly detection refers to identifying abnormal situations in systems and data through algorithms and rules. The lonely forest algorithm is used in the present invention. Regular security audits refer to regular inspections and assessments of the security status of systems and data to ensure compliance with security requirements, regular security scans, vulnerability assessments, permission audits and compliance checks, generation of security reports, and taking appropriate remedial measures based on the problems found.

[0023] According to the order of importance of electronic devices, the devices are divided into different data management priorities and security levels. The first A% of devices are assigned high priority, the middle B% of devices are assigned medium priority, and the last C% of devices are assigned low priority, where A, B and C are all user-defined positive integers.

[0024] According to step S400, the isolation forest model is trained using historical data to learn the patterns and features of normal data; the training data includes data on the operating status, performance indicators and usage of the electronic device; the trained isolation forest model is deployed to the real-time monitoring system, and the data of the electronic device is collected in real time through the 5G network and input into the isolation forest model for analysis; the real-time monitoring system continuously analyzes the data of the electronic device, and the isolation forest model scores each data point. When the data score of a certain device exceeds a preset threshold, the system determines that the data of the device is abnormal and triggers the abnormality handling process;

[0025] According to the importance sequence of key electronic equipment, the equipment is sorted according to the priority of importance; when the data of a certain electronic equipment is detected as abnormal, the system first records the abnormal situation of the equipment, and according to the importance sequence of the electronic equipment, the system prioritizes checking the data of other key equipment directly associated with the abnormal device; the system checks the data of high-priority equipment directly associated with the abnormal device to determine whether there are similar abnormal situations. If the data of the high-priority equipment is normal, its data is used to repair the data of the abnormal device; if the data of the high-priority associated equipment is also abnormal, the system continues to check the data of the medium-priority equipment associated with the abnormal device. If the data of the medium-priority equipment is normal, its data is used to repair the data of the abnormal device; if the data of the medium-priority associated equipment is also abnormal, the system finally checks the data of the low-priority equipment associated with the abnormal device. If the data of the low-priority equipment is normal, its data is used to repair the data of the abnormal device.

[0026] According to step S400, the remaining correct data of the electronic device collected is divided into training data and verification data, the training data is used to train the convolutional neural network, and the verification data is used to evaluate the accuracy of the model; the convolutional neural network structure is designed, and the network structure includes an input layer, a plurality of convolutional layers, a pooling layer, a fully connected layer, and an output layer; the convolutional neural network is trained using the prepared training data, and the accuracy and generalization ability of the model are optimized by continuously adjusting the network parameters and structure; during the training process, the verification data is used to evaluate the model performance to prevent overfitting; after the training is completed, the data of the abnormal device is input into the trained convolutional neural network model, and the model derives the correct data based on the remaining normal data, and the derived correct data is used to cover and repair the data of the abnormal device;

[0027] After the repair is completed, the system re-evaluates the repaired data. If problems are still found in the repaired data, the model needs to be adjusted and retrained.

[0028] An intelligent data management system based on 5G communication, comprising:

[0029] Data collection and preprocessing module: including: data collection unit, data preprocessing unit and data integration unit; wherein the data collection unit is responsible for collecting the operation data of all electronic devices of the company in real time through the 5G network, the operation data including the device status, performance indicators and usage; the data preprocessing unit is responsible for preprocessing the operation data; the data integration unit is responsible for integrating the preprocessed operation data into the electronic device data set;

[0030] Equipment association analysis module: including: association analysis unit and graph structure construction unit; wherein, the association analysis unit is responsible for analyzing the association between the company's electronic devices based on the electronic device data set, based on the data communication frequency and the similarity of data content, using the K-means algorithm and the Apriori algorithm, and the graph structure construction unit is responsible for representing the electronic devices and their association relationships as a graph structure to obtain the association parameters between the devices;

[0031] Level setting module: including: an associated parameter sorting unit, a redundant parameter removal unit and a priority and security level setting unit; wherein the associated parameter sorting unit arranges the associated parameters in descending order according to the breadth and strength of the association to form an importance sequence of the electronic equipment, the redundant parameter removal unit uses the Lasso regression algorithm to remove redundant parameters in the electronic equipment and retains the key parameters, and the priority and security level setting unit sets the data management priority and security level for each device according to the importance sequence of the electronic equipment, so that the important equipment can be managed with higher priority and have a higher level of security protection;

[0032] Anomaly detection and data repair module: includes: an anomaly detection unit, an associated device inspection unit and a data coverage and repair unit; wherein the anomaly detection unit uses an isolation forest algorithm to monitor electronic device data in real time and detect anomalies in the data. When an error occurs in the data of a certain electronic device, the associated device inspection unit is responsible for checking whether there is an error in the data of other devices associated with it according to the priority of the importance sequence of the electronic device. The data coverage and repair unit is responsible for covering and repairing the data. If there is no error in the data of the associated device, the correct data is used to cover the erroneous data. If there is an error in the data of all associated devices, the remaining correct data of the electronic device containing the erroneous data is used to deduce through a convolutional neural network to repair the data.

[0033] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0034] The present invention combines the K-means algorithm and the Apriori algorithm to accurately analyze the association between electronic devices, construct a graph structure between devices, and comprehensively capture the complex association relationships between devices.

[0035] The present invention uses the Lasso regression algorithm to remove redundant parameters according to the importance of associated parameters, retain key parameters, and intelligently set the data management priority and security level of the device to ensure that important devices are managed and protected with higher priority.

[0036] The present invention uses the isolation forest algorithm to monitor electronic device data in real time and detect anomalies in the data. When an anomaly is found, the data of the associated device is checked first and covered by the correct data of the associated device; if the data of all associated devices are wrong, the convolutional neural network is used to derive and repair the data to ensure the integrity and reliability of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 It is a schematic diagram of the steps of an intelligent data management method based on 5G communication of the present invention;

[0039] Figure 2 This is a system structure diagram of an intelligent data management system based on 5G communication in the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0042] According to an embodiment of the present invention, in a large enterprise, it is necessary to manage and monitor hundreds of electronic devices within the enterprise, including production equipment, servers and workstations. In order to verify the effectiveness of the system, 6 months of operation data are collected, including device status, performance indicators and usage. Figure 1 A step diagram of an intelligent data management method based on 5G communication is shown in the figure, an intelligent data management method based on 5G communication, the method includes the following steps:

[0043] S100, collecting operation data of all electronic devices of the company in real time through the 5G network, wherein the operation data includes device status, performance indicators and usage, preprocessing the operation data, and integrating the preprocessed operation data into an electronic device data set;

[0044] S200, according to the electronic device data set, based on the data communication frequency and the similarity of the data content, use the K-means algorithm and the Apriori algorithm to analyze the association between the electronic devices of the company; represent the electronic devices and their association relationships as a graph structure, and obtain the association parameters between the devices;

[0045] S300, according to the correlation parameters, the correlation breadth and strength are arranged in descending order to form an importance sequence of the electronic devices, and the Lasso regression algorithm is used to remove redundant parameters in the electronic devices and retain key parameters; according to the importance sequence of the electronic devices, a data management priority and a security level are set for each device, so that important devices can receive higher priority management and higher level security protection;

[0046] S400. Use the isolation forest algorithm to monitor the data of electronic devices in real time and detect anomalies in the data. When an error occurs in the data of a certain electronic device, check whether the data of other devices associated with it have errors according to the priority sequence of the importance of the electronic device; if the data of the associated device has no errors, overwrite the erroneous data with the correct data; if the data of all associated devices have errors, use the remaining correct data of the electronic device containing the erroneous data to deduce through a convolutional neural network and perform data repair.

[0047] According to step S100, the electronic device integrates a 5G communication module, uses the SSL / TLS encryption protocol to encrypt data before transmission, and uses the MQTT protocol as the data transmission protocol for transmission; uses Alibaba Cloud as the data receiving and storage center, and deploys a load balancer; designs a data receiving API interface to implement data writing operations, and uses Kafka as a message queue to buffer and distribute real-time data; cleans the received data at the data receiving and storage center, removes invalid data and duplicate data, and converts the data into a unified format; stores the preprocessed data in a PostgreSQL database, classifies and labels the data, and integrates the preprocessed operating data into an electronic device data set.

[0048] According to step S200, the data communication frequency between each pair of electronic devices is calculated. The higher the communication frequency, the higher the correlation between the two devices. The data communication frequency is obtained by weighted calculation by counting the number of communications and the amount of communication data between each pair of devices over a period of time. A pair of devices A and B communicate 1000 times and the amount of communication data is 500GB within 6 months. The data communication frequency is obtained by weighted calculation, where the maximum number of communications is 2000 times and the maximum amount of communication data is 1000GB. The formula for the communication frequency W is as follows:

[0049] ;

[0050] Analyze the data content of device status, performance indicators and usage, and calculate their similarity; compare the status change patterns of different devices to obtain status similarity, compare the performance indicators of different devices to obtain performance indicator similarity, and compare the usage patterns of devices to obtain usage similarity;

[0051] Use the K-means algorithm to group devices according to their features and get device clusters with similar communication frequency and content features. Use the Apriori algorithm to mine the association patterns between devices and find device combinations and association rules. Combine the communication frequency features and content similarity features to build a comprehensive feature vector, with each device represented by a feature vector. The similarity of the state change patterns of devices A and B is 0.8, the similarity of the performance indicators of devices A and B is 0.7, and the similarity of the usage of devices A and B is 0.9. Based on the above features, a comprehensive feature vector is built: .

[0052] When using the K-means algorithm to perform cluster analysis on feature vectors, three cluster centers are randomly initialized, the distance from each device to the cluster center is calculated, the device is assigned to the nearest cluster center, and the position of the cluster center is updated until the clustering results converge; the clustering results reflect the similarity and correlation between devices, and the devices in the same cluster have a high correlation; the clustering results show that the devices are divided into three groups according to their characteristics, and the correlation between devices in each group is high.

[0053] Use the Apriori algorithm to iteratively generate frequent item sets, calculate the frequency of each pair of devices, generate association rules, evaluate the support and confidence of the rules; and filter out the association rules with the highest support and confidence. The filtered association rules are as follows: Rule 1: The support between device A and device B is 0.8 and the confidence is 0.9. Rule 2: The support between device C and device D is 0.7 and the confidence is 0.85.

[0054] According to step S200, each electronic device is represented as a node, and each node contains basic information and features of the device; the edge represents the association relationship between the devices, and the weight of the edge represents the strength of the association between the devices; the direction of the edge allows directed edges and undirected edges, wherein directed edges represent unidirectional associations, and undirected edges represent bidirectional associations; a list containing all electronic device nodes is created, and an empty edge list is initialized to store the association relationships between the devices; the frequency of data communication between each pair of devices is counted, and the higher the frequency, the stronger the association; the data content of the device status, performance indicators and usage is compared, and their similarity is calculated, and the higher the similarity, the stronger the association; based on the calculated association, it is decided whether to add an edge between the nodes and the weight of the edge, and if the association between the two devices exceeds a certain threshold, an edge is added between the two devices and a weight is assigned;

[0055] Calculate the association parameters between devices, use Gephi to visualize the graph structure, and display the electronic devices and their association relationships; regularly recalculate the association parameters between devices so that the graph structure reflects the latest device association status; update the nodes and edges in the graph structure when adding new devices or removing faulty devices.

[0056] The association parameters between the devices include node degree, edge weight, adjacency matrix, node clustering coefficient, shortest path and node centrality; wherein the node degree is the number of edges directly connected to the node, reflecting the number of direct associations of the device. The higher the node degree, the wider the association of the device. The edge weight indicates the strength of the association between the devices. The larger the edge weight, the stronger the association between the devices. The edge weight is calculated comprehensively based on the communication frequency and content similarity. The adjacency matrix is ​​a square matrix. The elements of the matrix represent the association relationship between the nodes. The element value can be 0 or a weight value. 0 indicates no association, and a weight value indicates related The adjacency matrix is ​​used to query the direct association between devices. The clustering coefficient of a node represents the ratio of the number of actual edges between a node and its neighboring nodes to the number of possible edges. The clustering coefficient reflects the density of devices in the local network. The higher the coefficient, the stronger the association of the local network where the device is located. The shortest path is the path with the smallest weight among the paths connecting two nodes. The shortest path is used to measure the indirect association between devices. The shorter the path, the stronger the indirect association between devices. The node centrality is a measure of the importance of the node in the entire network, which is calculated through degree centrality, closeness centrality and betweenness centrality.

[0057] In this embodiment, the degree of device A is 5, indicating that it is connected to 5 devices; the edge weight between devices A and B is 0.5; the clustering coefficient of device A is 0.75; the shortest path length between devices A and C is 2; the degree centrality of device A is 0.05, the closeness centrality of device A is 0.8, and the betweenness centrality of device A is 0.2.

[0058] According to step S300, for each electronic device, a comprehensive importance score is calculated according to the breadth and strength of its associated parameters, and the importance of each parameter is weighted and summed to obtain a comprehensive importance score for each device; all electronic devices are arranged in descending order according to the comprehensive importance score to form an importance sequence of the electronic devices;

[0059] The importance score of the equipment is used as the independent variable, and other business-related target variables are used as dependent variables to construct a data set; the data is standardized to ensure that the numerical range of all parameters is consistent, and the data set is divided into a training set and a test set; the Lasso regression algorithm is used for model training, and by adding absolute value regularization terms to the loss function, the coefficients of certain parameters tend to zero, thereby achieving parameter selection and dimensionality reduction; during the training process, Lasso regression will gradually reduce the coefficient values ​​of unimportant parameters to zero to remove redundant parameters; after the training is completed, check the coefficient values ​​of each parameter in the model, and retain the parameters with non-zero coefficient values ​​as the key parameters that have a greater impact on the target variable; the parameters that are reduced to zero are redundant parameters and are removed; use the test set to evaluate the model performance, and adjust and optimize the parameter selection based on the model evaluation results.

[0060] According to step S300, the data management priority is divided into three levels: high, medium and low, wherein the high priority requires the most real-time and most frequent data collection and processing, the medium priority requires regular data collection and processing, and the low priority performs batch data collection and processing; the security level is divided into three levels: high, medium and low, wherein the high security level requires the highest level of security protection, including encrypted transmission, storage encryption, access control, real-time monitoring and anomaly detection, the medium security level requires medium-level security protection, including encrypted transmission, storage encryption, regular security audits and anomaly detection, and the low security level requires basic security protection, including encrypted transmission, access control and anomaly detection;

[0061] According to the order of importance of electronic devices, the devices are divided into different data management priorities and security levels, with the first 20% of devices assigned high priority, the middle 60% of devices assigned medium priority, and the bottom 20% of devices assigned low priority.

[0062] According to step S400, the isolation forest model is trained using historical data to learn the patterns and features of normal data; the training data includes data on the operating status, performance indicators and usage of the electronic device; the trained isolation forest model is deployed to the real-time monitoring system, and the data of the electronic device is collected in real time through the 5G network and input into the isolation forest model for analysis; the real-time monitoring system continuously analyzes the data of the electronic device, and the isolation forest model scores each data point. When the data score of a certain device exceeds a preset threshold, the system determines that the data of the device is abnormal and triggers the abnormality handling process;

[0063] According to the importance sequence of key electronic equipment, the equipment is sorted according to the priority of importance; when the data of a certain electronic equipment is detected as abnormal, the system first records the abnormal situation of the equipment, and according to the importance sequence of the electronic equipment, the system prioritizes checking the data of other key equipment directly associated with the abnormal device; the system checks the data of high-priority equipment directly associated with the abnormal device to determine whether there are similar abnormal situations. If the data of the high-priority equipment is normal, its data is used to repair the data of the abnormal device; if the data of the high-priority associated equipment is also abnormal, the system continues to check the data of the medium-priority equipment associated with the abnormal device. If the data of the medium-priority equipment is normal, its data is used to repair the data of the abnormal device; if the data of the medium-priority associated equipment is also abnormal, the system finally checks the data of the low-priority equipment associated with the abnormal device. If the data of the low-priority equipment is normal, its data is used to repair the data of the abnormal device.

[0064] According to step S400, the remaining correct data of the electronic device collected is divided into training data and verification data, the training data is used to train the convolutional neural network, and the verification data is used to evaluate the accuracy of the model; the convolutional neural network structure is designed, and the network structure includes an input layer, a plurality of convolutional layers, a pooling layer, a fully connected layer, and an output layer; the convolutional neural network is trained using the prepared training data, and the accuracy and generalization ability of the model are optimized by continuously adjusting the network parameters and structure; during the training process, the verification data is used to evaluate the model performance to prevent overfitting; after the training is completed, the data of the abnormal device is input into the trained convolutional neural network model, and the model derives the correct data based on the remaining normal data, and the derived correct data is used to cover and repair the data of the abnormal device;

[0065] After the repair is completed, the system re-evaluates the repaired data. If problems are still found in the repaired data, the model needs to be adjusted and retrained.

[0066] According to another embodiment of the present invention, Figure 2 As shown in the system structure diagram of an intelligent data management system based on 5G communication, an intelligent data management system based on 5G communication includes:

[0067] Data collection and preprocessing module: including: data collection unit, data preprocessing unit and data integration unit; wherein the data collection unit is responsible for collecting the operation data of all electronic devices of the company in real time through the 5G network, the operation data including the device status, performance indicators and usage; the data preprocessing unit is responsible for preprocessing the operation data; the data integration unit is responsible for integrating the preprocessed operation data into the electronic device data set;

[0068] Equipment association analysis module: including: association analysis unit and graph structure construction unit; wherein, the association analysis unit is responsible for analyzing the association between the company's electronic devices based on the electronic device data set, based on the data communication frequency and the similarity of data content, using the K-means algorithm and the Apriori algorithm, and the graph structure construction unit is responsible for representing the electronic devices and their association relationships as a graph structure to obtain the association parameters between the devices;

[0069] Level setting module: including: an associated parameter sorting unit, a redundant parameter removal unit and a priority and security level setting unit; wherein the associated parameter sorting unit arranges the associated parameters in descending order according to the breadth and strength of the association to form an importance sequence of the electronic equipment, the redundant parameter removal unit uses the Lasso regression algorithm to remove redundant parameters in the electronic equipment and retains the key parameters, and the priority and security level setting unit sets the data management priority and security level for each device according to the importance sequence of the electronic equipment, so that the important equipment can be managed with higher priority and have a higher level of security protection;

[0070] Anomaly detection and data repair module: includes: an anomaly detection unit, an associated device inspection unit and a data coverage and repair unit; wherein the anomaly detection unit uses an isolation forest algorithm to monitor electronic device data in real time and detect anomalies in the data. When an error occurs in the data of a certain electronic device, the associated device inspection unit is responsible for checking whether there is an error in the data of other devices associated with it according to the priority of the importance sequence of the electronic device. The data coverage and repair unit is responsible for covering and repairing the data. If there is no error in the data of the associated device, the correct data is used to cover the erroneous data. If there is an error in the data of all associated devices, the remaining correct data of the electronic device containing the erroneous data is used to deduce through a convolutional neural network to repair the data.

[0071] In this embodiment, the anomaly detection unit uses the isolation forest algorithm to perform anomaly detection. The data score of device A is 0.95, which exceeds the preset threshold value of 0.9 and is determined to be abnormal. The data score of device B is 0.85, which does not exceed the preset threshold value of 0.9 and is determined to be normal. The associated device inspection unit checks the data of other devices associated with device A. The priority of device A in the importance sequence is 1, and the associated devices are as follows: device B, device C, and device D; check the data of device B, device C, and device D: the data score of device B is 0.85 (normal), the data score of device C is 0.92 (abnormal), and the data score of device D is 0.88 (normal). The detection results are as follows: the data of device A and device C are abnormal. The data of device B and device D are normal.

[0072] The data overwriting and repair unit performs data overwriting, and the data of device A is overwritten by the normal data of device B and device D, and the relevant data of device B and device D are copied to the corresponding data position of device A. The remaining data of device A and device C are all wrong, and the remaining correct data of the electronic device containing the wrong data is used to deduce through the convolutional neural network. After repair, the data of device A and device C are judged to be normal after repair, with scores of 0.85 and 0.87 respectively.

[0073] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0074] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent data management method based on 5G communication, characterized in that: The method comprises the following steps: S100, collecting operation data of all electronic devices of the company in real time through the 5G network, wherein the operation data includes device status, performance indicators and usage, preprocessing the operation data, and integrating the preprocessed operation data into an electronic device data set; S200, according to the electronic device data set, based on the data communication frequency and the similarity of the data content, use the K-means algorithm and the Apriori algorithm to analyze the association between the electronic devices of the company; represent the electronic devices and their association relationships as a graph structure, and obtain the association parameters between the devices; S300, calculating a comprehensive importance score according to the breadth and strength of the associated parameters, and weighting and summing the importance of each parameter to obtain a comprehensive importance score for each device; Arrange all electronic devices in descending order according to their comprehensive importance scores to form an importance sequence of electronic devices; Use Lasso regression algorithm to remove redundant parameters in electronic devices and retain key parameters; set data management priority and security level for each device according to the importance sequence of electronic devices, so that important devices can receive higher priority management and higher level of security protection; According to step S300, for each electronic device, a comprehensive importance score is calculated according to the breadth and strength of its associated parameters, and the importance of each parameter is weighted and summed to obtain a comprehensive importance score for each device; all electronic devices are arranged in descending order according to the comprehensive importance score to form an importance sequence of the electronic devices; The comprehensive importance score of the equipment is used as the independent variable, and other business-related target variables are used as dependent variables to construct a data set; the data is standardized to ensure that the numerical range of all parameters is consistent, and the data set is divided into a training set and a test set; The Lasso regression algorithm is used for model training. By adding absolute value regularization terms to the loss function, the coefficients of certain parameters tend to zero, thereby achieving parameter selection and dimensionality reduction. During the training process, Lasso regression will gradually reduce the coefficient values ​​of unimportant parameters to zero to remove redundant parameters; After the training is completed, check the coefficient values ​​of each parameter in the model, and retain the parameters with non-zero coefficient values ​​as the key parameters that have a greater impact on the target variable; the parameters that are reduced to zero are redundant parameters and are removed; Use the test set to evaluate model performance and adjust and optimize parameter selection based on the model evaluation results; S400, using the isolation forest algorithm to monitor the electronic device data in real time and detect abnormalities in the data. When an error occurs in the data of a certain electronic device, check whether there is an error in the data of other devices associated with it according to the priority of the electronic device's importance sequence; If the data of the associated device is error-free, the erroneous data is overwritten with the correct data; if the data of all associated devices are erroneous, the remaining correct data of the electronic device containing the erroneous data is used to deduce through a convolutional neural network to repair the data.

2. According to claim 1, an intelligent data management method based on 5G communication is characterized in that: According to step S100, the electronic device integrates a 5G communication module, encrypts data using the SSL / TLS encryption protocol before transmission, and uses the MQTT protocol as the data transmission protocol for transmission; uses Alibaba Cloud as the data receiving and storage center, and deploys a load balancer; designs a data receiving API interface to implement data writing operations, and uses Kafka as a message queue to buffer and distribute real-time data; cleans the received data in the data receiving and storage center, removes invalid data and duplicate data, and converts the data into a unified format; The preprocessed data is stored in the PostgreSQL database, the data is classified and labeled, and the preprocessed operation data is integrated into the electronic equipment data set.

3. The intelligent data management method based on 5G communication according to claim 1, characterized in that: According to step S200, the data communication frequency between each pair of electronic devices is calculated. The higher the communication frequency, the higher the correlation between the two devices. The data communication frequency is obtained by weighted calculation by counting the number of communications and the amount of communication data between each pair of devices over a period of time; the data content of the device status, performance indicators and usage is analyzed to calculate their similarity; Compare the state change patterns of different devices to obtain state similarity, compare the performance indicators of different devices to obtain performance indicator similarity, and compare the usage patterns of devices to obtain usage similarity; Use the K-means algorithm to group devices according to their characteristics and obtain device clusters with similar communication frequency and content characteristics; Use the Apriori algorithm to mine the association patterns between devices and find device combinations and association rules. Combine the communication frequency characteristics and content similarity characteristics to construct a comprehensive feature vector, with each device represented by a feature vector. When using the K-means algorithm to perform cluster analysis on feature vectors, several cluster centers are randomly initialized, the distance from each device to the cluster center is calculated, the device is assigned to the nearest cluster center, and the location of the cluster center is updated until the clustering results converge; The clustering results reflect the similarity and correlation between devices. Devices in the same cluster have a high correlation. Use the Apriori algorithm to iteratively generate frequent item sets, calculate the frequency of occurrence of each pair of devices, generate association rules, and evaluate the support and confidence of the rules; Filter out the association rules with the highest support and confidence.

4. The intelligent data management method based on 5G communication according to claim 3 is characterized in that: According to step S200, each electronic device is represented as a node, and each node contains the characteristics of the device; the edge represents the association relationship between the devices, and the weight of the edge represents the strength of the association between the devices; the direction of the edge allows directed edges and undirected edges, wherein directed edges represent unidirectional associations, and undirected edges represent bidirectional associations; a list containing all electronic device nodes is created, and an empty edge list is initialized to store the association relationships between the devices; the frequency of data communication between each pair of devices is counted, and the higher the frequency, the stronger the association; the data content of the device status, performance indicators and usage is compared, and their similarity is calculated, and the higher the similarity, the stronger the association; based on the calculated association, it is decided whether to add an edge between the nodes and the weight of the edge, and if the association between the two devices exceeds a certain threshold, an edge is added between the two devices and a weight is assigned; Calculate the association parameters between devices, use Gephi to visualize the graph structure, and display the electronic devices and their association relationships; regularly recalculate the association parameters between devices so that the graph structure reflects the latest device association status; update the nodes and edges in the graph structure when adding new devices or removing faulty devices.

5. The intelligent data management method based on 5G communication according to claim 4 is characterized in that: The association parameters between the devices include node degree, edge weight, adjacency matrix, node clustering coefficient, shortest path and node centrality; wherein the node degree is the number of edges directly connected to the node, reflecting the number of direct associations of the device. The higher the node degree, the wider the association of the device. The edge weight indicates the strength of the association between the devices. The larger the edge weight, the stronger the association between the devices. The edge weight is calculated comprehensively based on the communication frequency and content similarity. The adjacency matrix is ​​a square matrix. The elements of the matrix represent the association relationship between the nodes. The element value can be 0 or a weight value. 0 indicates no association, and a weight value indicates related The adjacency matrix is ​​used to query the direct association between devices. The clustering coefficient of a node represents the ratio of the number of actual edges between a node and its neighboring nodes to the number of possible edges. The clustering coefficient reflects the density of devices in the local network. The higher the coefficient, the stronger the association of the local network where the device is located. The shortest path is the path with the smallest weight among the paths connecting two nodes. The shortest path is used to measure the indirect association between devices. The shorter the path, the stronger the indirect association between devices. The node centrality is a measure of the importance of the node in the entire network, which is calculated through degree centrality, closeness centrality and betweenness centrality.

6. The intelligent data management method based on 5G communication according to claim 1, characterized in that: According to step S300, the data management priority is divided into three levels: high, medium and low, wherein the high priority requires the most real-time and most frequent data collection and processing, the medium priority requires regular data collection and processing, and the low priority performs batch data collection and processing; the security level is divided into three levels: high, medium and low, wherein the high security level requires the highest level of security protection, including encrypted transmission, storage encryption, access control, real-time monitoring and anomaly detection, the medium security level requires medium-level security protection, including encrypted transmission, storage encryption, regular security audits and anomaly detection, and the low security level requires basic security protection, including encrypted transmission, access control and anomaly detection; According to the order of importance of electronic devices, the devices are divided into different data management priorities and security levels. The first A% of devices are assigned high priority, the middle B% of devices are assigned medium priority, and the last C% of devices are assigned low priority, where A, B and C are all user-defined positive integers.

7. The intelligent data management method based on 5G communication according to claim 1, characterized in that: According to step S400, the isolation forest model is trained using historical data to learn the patterns and features of normal data; The training data includes the operating status, performance indicators and usage data of electronic devices. The trained isolation forest model is deployed to the real-time monitoring system, and the data of electronic devices is collected in real time through the 5G network and input into the isolation forest model for analysis. The real-time monitoring system continuously analyzes the data of electronic devices, and the isolation forest model scores each data point. When the data score of a device exceeds the preset threshold, the system determines that the device data is abnormal and triggers the abnormality handling process. According to the importance sequence of electronic devices, the devices are sorted according to the priority of importance; when the data of a certain electronic device is detected as abnormal, the system first records the abnormal situation of the device, and according to the importance sequence of electronic devices, the system prioritizes checking the data of other devices directly associated with the abnormal device; the system checks the data of high-priority devices directly associated with the abnormal device to determine whether there are similar abnormal situations. If the data of the high-priority device is normal, its data is used to repair the data of the abnormal device; if the data of the high-priority associated device is also abnormal, the system continues to check the data of the medium-priority device associated with the abnormal device. If the data of the medium-priority device is normal, its data is used to repair the data of the abnormal device; if the data of the medium-priority associated device is also abnormal, the system finally checks the data of the low-priority device associated with the abnormal device. If the data of the low-priority device is normal, its data is used to repair the data of the abnormal device.

8. The intelligent data management method based on 5G communication according to claim 7, characterized in that: According to step S400, the remaining correct data of the electronic device collected is divided into training data and verification data, the training data is used to train the convolutional neural network, and the verification data is used to evaluate the accuracy of the model; the convolutional neural network structure is designed, and the network structure includes an input layer, a plurality of convolutional layers, a pooling layer, a fully connected layer, and an output layer; the convolutional neural network is trained using the prepared training data, and the accuracy and generalization ability of the model are optimized by continuously adjusting the network parameters and structure; during the training process, the verification data is used to evaluate the model performance to prevent overfitting; after the training is completed, the data of the abnormal device is input into the trained convolutional neural network model, and the model derives the correct data based on the remaining normal data, and the derived correct data is used to cover and repair the data of the abnormal device; After the repair is completed, the system re-evaluates the repaired data. If problems are still found in the repaired data, the model needs to be adjusted and retrained.

9. An intelligent data management system based on 5G communication, using an intelligent data management method based on 5G communication according to any one of claims 1 to 8, characterized in that: include: Data collection and preprocessing module: including: data collection unit, data preprocessing unit and data integration unit; wherein the data collection unit is responsible for collecting the operation data of all electronic devices of the company in real time through the 5G network, the operation data including the device status, performance indicators and usage; the data preprocessing unit is responsible for preprocessing the operation data; the data integration unit is responsible for integrating the preprocessed operation data into the electronic device data set; Equipment association analysis module: including: association analysis unit and graph structure construction unit; wherein, the association analysis unit is responsible for analyzing the association between the company's electronic devices based on the electronic device data set, based on the data communication frequency and the similarity of data content, using the K-means algorithm and the Apriori algorithm, and the graph structure construction unit is responsible for representing the electronic devices and their association relationships as a graph structure to obtain the association parameters between the devices; Level setting module: including: associated parameter sorting unit, redundant parameter removal unit and priority and security level setting unit; wherein, the associated parameter sorting unit calculates the comprehensive importance score according to the breadth and strength of the associated parameters, and weights and sums the importance of each parameter to obtain the comprehensive importance score of each device; all electronic devices are arranged in descending order according to the comprehensive importance score to form an importance sequence of electronic devices; the redundant parameter removal unit uses the Lasso regression algorithm to remove redundant parameters in the electronic devices and retains key parameters; the priority and security level setting unit sets the data management priority and security level for each device according to the importance sequence of the electronic devices, so that important devices can receive higher priority management and higher level of security protection; during the training process, the Lasso regression will gradually reduce the coefficient value of unimportant parameters to zero to remove redundant parameters; after the training is completed, check the coefficient value of each parameter in the model, and retain the parameters with non-zero coefficient values ​​as key parameters that have a greater impact on the target variable; the parameters that are reduced to zero are redundant parameters and are removed; use the test set to evaluate the model performance, and adjust and optimize the parameter selection according to the model evaluation results; Anomaly detection and data repair module: includes: an anomaly detection unit, an associated device inspection unit and a data coverage and repair unit; wherein the anomaly detection unit uses an isolation forest algorithm to monitor electronic device data in real time and detect anomalies in the data. When an error occurs in the data of a certain electronic device, the associated device inspection unit is responsible for checking whether there is an error in the data of other devices associated with it according to the priority of the importance sequence of the electronic device. The data coverage and repair unit is responsible for covering and repairing the data. If there is no error in the data of the associated device, the correct data is used to cover the erroneous data. If there is an error in the data of all associated devices, the remaining correct data of the electronic device containing the erroneous data is used to deduce through a convolutional neural network to repair the data.

Images