Data management method, device, system and readable storage medium

Abstract

The present disclosure relates to the field of computer technologies, and relates to a data management method, apparatus, system, and readable storage medium. The data management method is applied to a first server and includes: in response to the normal communication status of a first network, obtaining operation update data of a target device through a second server; and storing the operation update data through a distributed message service queue system to update historical operation data that is pre-stored and corresponds to the target device based on the operation update data; in response to the communication status of the first network returning to normal, determining unprocessed information to be consumed in the distributed message service queue system during the disconnection of the first network; and accessing the operation data based on the information to be consumed. It can ensure the accuracy and integrity of the obtained operation data, and when the communication status of the first network returns to normal, it can also ensure the timeliness of processing the information to be consumed, thereby facilitating performance improvement and subsequent targeted management and maintenance of the target device.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a data management method, a data management device, a data management system, and a computer-readable storage medium. Background Art

[0002] With the continuous development of industrial automation, the real-time performance and reliability of data transmission have become important indicators for measuring system performance. In related technologies, the data information of a device is directly obtained from the device through network transmission. However, this data transmission method depends on a single data storage and transmission path. Therefore, when the network experiences an abnormality during the data transmission process, the integrity of the obtained data information will be affected, thus restricting the intelligent level of the industrial production process. Summary of the Invention

[0003] To overcome the problems existing in related technologies, an exemplary embodiment of the present disclosure discloses a data management method applied to a first server, including: in response to the normal communication status of a first network, obtaining the running update data of a target device through a second server, where the second server is connected to the target device through a second network and is used to record and save the current running data of the target device, and the running update data is determined based on the current running data, and the stability of the second network is higher than that of the first network; and storing the running update data through a distributed message service queue system to update the pre-stored historical running data corresponding to the target device based on the running update data; in response to the communication status of the first network returning to normal, determining the unprocessed information to be consumed in the distributed message service queue system during the disconnection of the first network, where the information to be consumed is the information to be accessed and interacted with the running data in the distributed message service queue system, and the running data is the data obtained after the last update of the distributed message service queue system before the communication status of the first network is disconnected; and accessing the running data based on the information to be consumed.

[0004] In some embodiments, in response to the normal communication status of the first network, obtaining the running update data of the target device through the second server includes: receiving the running update data of the target device sent by the second server, where the second server is determined based on the detection result of the data operation log of the target database, the target database is deployed in the second server and corresponds to the target device, and the target database is used to record and save the current running data of the target device.

[0005] In some embodiments, the running update data is stored through a distributed message service queue system to update the pre-stored historical running data corresponding to the target device based on the running update data, including: performing a first encryption process on the running update data to obtain a first encryption result; storing the first encryption result through the distributed message service queue system, so that the distributed message service queue system updates the pre-stored historical running data corresponding to the target device based on the first decryption result of the first encryption result to obtain an update result, where the first decryption process corresponding to the first decryption result corresponds to the first encryption process; caching the update result in the distributed message service queue system through the target search engine.

[0006] In some embodiments, the distributed message service queue system includes multiple intermediate storage nodes; storing the first encryption result through the distributed message service queue system includes: storing the first encryption result through the multiple intermediate storage nodes respectively.

[0007] In some embodiments, caching the update result in the distributed message service queue system includes: performing a second encryption process on the update result through the distributed message service queue system to obtain a second encryption result, and sending the second encryption result to the target search engine; performing a second decryption process on the second encryption result through the target search engine to cache the update result according to the obtained second decryption result, where the second decryption process corresponds to the second encryption result.

[0008] In some embodiments, in response to the communication status of the first network returning to normal, determining the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network includes: in response to the communication status of the first network returning to normal, regularly taking snapshots of the distributed message service queue system through the target search engine to determine the unprocessed information to be consumed by the distributed message service queue system during the interruption of the first network.

[0009] In some embodiments, the method further includes: obtaining the content update data of the target device from the file management server, where the content update data is obtained by the file management server from the second server, and the file management server is connected to the first server and the second server through the first network; caching the content update data through the target search engine for access by the client of the target search engine.

[0010] In some embodiments, the method further includes: displaying the running data through a data visualization interface.

[0011] Second aspect, the present disclosure also provides a data management device, which is applied to the first server and includes: an acquisition module, configured to obtain the running update data of the target device through the second server in response to the normal communication status of the first network, where the second server is connected to the target device through the second network and is used to record and save the current running data of the target device, and the running update data is determined based on the current running data, and the stability of the second network is higher than that of the first network; a storage module, configured to store the running update data through a distributed message service queue system to update the pre-stored historical running data corresponding to the target device based on the running update data; a first processing module, configured to determine the unprocessed information to be consumed in the distributed message service queue system during the disconnection of the first network in response to the restoration of the normal communication status of the first network, where the information to be consumed is the information to be accessed and interacted with the running data in the distributed message service queue system, and the running data is the data obtained after the last update before the disconnection of the communication status of the first network in the distributed message service queue system; a second processing module, configured to access the running data based on the information to be consumed.

[0012] Third aspect, the present disclosure also provides a data management system, including: a target device; a second server, connected to the target device through the second network and used to record and save the current running data of the target device; a first server, connected to the second server through the first network, and the stability of the second network is higher than that of the first network. The first server is used to obtain the running update data of the target device through the second server, store the running update data through a distributed message service queue system to update the pre-stored historical running data corresponding to the target device based on the running update data, determine the unprocessed information to be consumed in the distributed message service queue system during the disconnection of the first network, and access the running data based on the information to be consumed, where the running update data is determined based on the current running data, the information to be consumed is the information to be accessed and interacted with the running data in the distributed message service queue system, and the running data is the data obtained after the last update before the disconnection of the communication status of the first network in the distributed message service queue system.

[0013] In some embodiments, the distributed message service queue system includes multiple intermediate storage nodes, which are respectively deployed on corresponding physical servers.

[0014] In some embodiments, the first server includes a target search engine, and the target search engine is used to cache the running data in the distributed message service queue system.

[0015] In some embodiments, the data management system further includes: a file management server connected to the second server and the first server respectively through a first network. The file management server is configured to obtain content update data of the target device from the second server and send the content update data to the target search engine, so as to cache the content update data through the target search engine for access by the client of the target search engine.

[0016] In a fourth aspect, the present disclosure also provides a computer-readable storage medium storing the following program, and the program is used to execute the data management method according to any of the above embodiments.

[0017] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: According to the data management method provided by the present disclosure, a connection between the first server and the second server is established in advance through the first network, and a connection between the second server and the target device is established through the second network, and the stability of the second network is higher than that of the first network, which can ensure that the operation data generated during the operation of the target device can be completely obtained by the second server. Furthermore, when the communication status of the first network is normal, the first server can timely obtain the operation update data of the target device from the second server and store it through the distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data, so that the operation data corresponding to the target device stored in the first server can be updated in a timely manner and can correspond to the actual operation situation of the target device, which helps to ensure the accuracy and integrity of the data. And when it is detected that the communication status of the first network resumes normal, by determining the unprocessed messages to be consumed in the distributed message service queue system during the disconnection of the first network, it can be clear which messages need to be processed in a timely manner, and then access the operation data based on the messages to be consumed, which can ensure the timeliness of processing the messages to be consumed, ensure the integrity and reliability of the operation data, and thus is beneficial to improving the performance of the first server and facilitating subsequent targeted management and maintenance of the target device.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention can be better understood by describing the exemplary embodiments of the present invention in conjunction with the accompanying drawings. In the drawings:

[0020] Figure 1 is a schematic diagram of the architecture of a data management system shown according to an exemplary embodiment of the present disclosure;

[0021] Figure 2 is a schematic flowchart of a data management method shown according to an exemplary embodiment of the present disclosure;

[0022] Figure 3 is a schematic flowchart of another data management method shown according to an exemplary embodiment of the disclosure;

[0023] Figure 4 is a schematic flowchart of yet another data management method shown according to an exemplary embodiment of the disclosure;

[0024] Figure 5 is a schematic architecture diagram of another data management system shown according to an exemplary embodiment of the disclosure;

[0025] Figure 6 is a schematic architecture diagram of yet another data management system shown according to an exemplary embodiment of the disclosure;

[0026] Figure 7 is a schematic diagram of the structure of a data management device shown according to an exemplary embodiment of the disclosure. Specific Embodiments

[0027] Specific embodiments of the present invention will be described below. It should be noted that in the specific description of these embodiments, for the sake of concise description, this specification may not describe all features of the actual embodiments in detail. It should be understood that in the actual implementation of any embodiment, just as in the process of any engineering project or design project, in order to achieve the specific goals of the developer and to meet system-related or business-related constraints, various specific decisions are often made, and these may vary from one embodiment to another. In addition, it should also be understood that although the efforts made in such a development process may be complex and time-consuming, for those of ordinary skill in the art related to the content disclosed in the present invention, some design, manufacturing, or production changes based on the technical content disclosed in this disclosure are only conventional technical means and should not be understood as insufficient disclosure of the present invention.

[0028] Unless otherwise defined, technical terms or scientific terms used in the claims and the specification shall have the ordinary meanings understood by those of ordinary skill in the technical field to which the present invention pertains. The terms "first", "second" and similar terms used in the specification and claims of this patent application for invention do not denote any order, quantity or importance, but are only used to distinguish different components. The terms such as "a" or "an" do not denote a limitation of quantity, but mean that there is at least one. The terms such as "comprising" or "including" mean that the elements or items appearing before "comprising" or "including" cover the elements or items listed after "comprising" or "including" and their equivalent elements, and do not exclude other elements or items. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, nor are they limited to direct or indirect connections.

[0029] In the related art, the operation data of a device is directly obtained from the device through network transmission. That is, an industrial Internet device is pre-connected to a device server deployed in the same network environment, and then based on the communication connection between the two, the device server obtains and stores the data information of the industrial Internet device.

[0030] This data transmission method relies on a single data storage and transmission path. When an abnormality occurs in the network connection between the industrial Internet device and the device server during data transmission (for example: network interruption, congestion, delay, etc.), it will affect the integrity of the data information obtained by the device server, thereby restricting the intelligent level of the industrial production process.

[0031] To solve the above problems, an exemplary embodiment of the present disclosure provides a data management system, as Figure 1 shown, the data management system 100 includes: a target device 110, a second server 120, and a first server 130.

[0032] Among them, the target device 110 can be any industrial Internet device capable of network communication. For example, the target device 110 may include, but is not limited to, intelligent ore dressing machines, numerically controlled machine tools, industrial robots, etc., and can be specifically determined according to actual data management requirements.

[0033] The second server 120 is a server capable of establishing communication connections with the target device 110 and the first server 130 in different network environments. Based on the communication connection with the target device 110, the second server 120 can obtain and store the operation data of the target device 110, and based on the communication connection with the first server 130, provide the operation data of the target device 110 for the first server 130, so that the first server 130 can perform targeted maintenance on the target device 110 based on the obtained operation data. The second server 120 can be a highly available server, so that even in case of failures or anomalies during the connection with the target device 110 and the first server 130, it can continue to provide services. For example, the second server 120 can be a Cluster of Database Servers (CDS), a hybrid storage server, a backup server, etc. Among them, CDS is a cluster system formed by combining multiple database servers, which can improve the availability, reliability, and performance of data by distributing data across multiple servers.

[0034] In some examples, the second server 120 can be CDS. Each server in CDS can obtain and store the operation data of the target device 110, and provide consistent services by sharing data and coordinating work. When the current server fails or is abnormal, other servers can take over its work to ensure the continuity of services, thus effectively reducing the risk of service interruption, effectively ensuring the stability of data transmission between the target device 110 and CDS, and ensuring the integrity of the obtained operation data.

[0035] In some other examples, the data transmission between the second server 120 and the target device 110 can be carried out in a wired connection manner, which can ensure the stability and reliability of data transmission, reduce the risk of data loss and interference. At the same time, the wired connection can also provide a higher transmission speed and lower latency, which is suitable for scenarios with high requirements for data transmission.

[0036] The first server 130 can be a device server for storing and managing the operation data of the target device 110. The first server 130 can include, but is not limited to, the following multiple features: having multiple interfaces, supporting multiple communication protocols, having a certain data processing capacity, and being able to ensure data transmission security, etc. Among them, the multiple interfaces can include: Ethernet, serial port, USB, etc., so as to be able to connect different types of target devices 110 or the second server 120 according to requirements. The types of communication protocols can include, but are not limited to: Transmission Control Protocol / Internet Protocol (TCP / IP), Modbus TCP (an industrial automation communication protocol based on the TCP / IP protocol), UDP (a connectionless network protocol), HTTP (an application layer protocol for the World Wide Web), etc., so as to be able to communicate with different types of target devices 110 or the second server 120.

[0037] In some optional implementation scenarios, the second server 120 and the first server 130 are connected through a first network, and the target device 110 and the second server 120 are connected through a second network. By adopting different network environments and controlling the second server 120 to establish connections with the first server 130 and the target device 110 respectively, it helps to enhance the reliability of the network environment, is conducive to achieving network environment isolation, ensures the transmission security of the operation data of the target device 110, and thus helps to meet various data management requirements.

[0038] Based on the same inventive concept, an exemplary embodiment of the present disclosure provides a data management method, which is applied to the first server in the above data management system. As Figure 2 shown, the data management method can include the following steps:

[0039] Step S210, in response to the normal communication status of the first network, obtain the running update data of the target device through the second server.

[0040] The first network is the network through which the first server communicates with the second server. If the communication status of the first network is normal, it indicates that the communication status between the first server and the second server is in a normal state. The installation location of the target device depends on its actual application requirements. Therefore, to ensure the stability of the second server in obtaining the current operating data of the target device, the second server is controlled to connect to the target device through the second network to record and save the current operating data of the target device. By using different networks to control the connection between the second server and the first server and the target device respectively, it helps to enhance the reliability of the network environment, is conducive to achieving network environment isolation, and ensures the transmission security of the operating data of the target device. Among them, the stability of the second network is higher than that of the first network. Therefore, when data is transmitted in the network environment provided by the second network, the integrity of the current operating data obtained by the second server can be guaranteed.

[0041] Since the second server is used to record and save the current operating data of the target device, the second server includes complete data information of the target device during operation. In response to the normal communication status of the first network, it indicates that the first server and the second server can communicate normally. However, during the operation of the target device, its generated operating data will be continuously updated and changed. Therefore, to improve data transmission efficiency and reduce redundant data, the current operating data of the target device obtained by the second server is matched with the existing historical operating data to determine the new operating update data generated during the operation of the target device, and the operating update data is obtained. That is, when the first network is normal, the first server can, based on the connection with the second server, timely obtain the latest operating data of the target device, enabling the first server to monitor and manage the operating status of the target device in real time, which helps to improve the reliability and stability of the data management system, facilitates timely discovery and resolution of problems, and ensures the normal operation of the target device. Among them, the operating update data may include, but is not limited to, data such as the status information, performance indicators, and configuration parameters of the target device.

[0042] In some optional examples, the first network can be the external network, and the second network can be the internal network. The external network, also known as the Internet, refers to a computer network that is interconnected globally, with a large network scope and high openness. Establishing a communication connection between the first server and the second server through the external network can improve the flexibility of the connection and meet various data management requirements. The internal network, also known as the local area network, is a computer network that can be interconnected within an enterprise or organization, with a certain degree of privacy and security. By controlling the connection between the target device and the second server through the internal network and using the internal network for data transmission, the stability and security of data transmission can be guaranteed, the risk of service interruption can be effectively reduced, and the integrity of the obtained operating data can be ensured.

[0043] In some optional implementation scenarios, taking the target device as an industrial Internet of Things device as an example, in the network environment provided by the second network, the second server can establish a connection with the target device through the Modbus TCP protocol, thereby achieving the purpose of real-time monitoring of the operating status of the target device. In some optional examples, if the target device includes multiple sub-industrial Internet of Things devices, the second server can establish connections with each sub-industrial Internet of Things device through the Modbus TCP protocol respectively, so that the second server can monitor the operating status of each sub-industrial Internet of Things device in real time. Then, when managing and maintaining the target device based on the obtained operation data subsequently, the operating status of each sub-industrial Internet of Things device in the target device can be better understood, which is convenient for improving the problem location efficiency and enhancing the performance of the target device, thereby enabling better maintenance and management of the target device. For example, the target device is an ore separator, and the multiple sub-industrial Internet of Things devices it includes can include, but are not limited to, sub-industrial Internet of Things devices such as Programmable Logic Controllers (PLCs), industrial control computers, injection devices, and air conditioners. To determine the operating status of the ore separator, the second server will be controlled to connect to sub-industrial Internet of Things devices such as PLCs, industrial control computers, injection devices, and air conditioners respectively through the Modbus TCP protocol. Then, during the operation of the ore separator, the operation data of each sub-industrial Internet of Things device can be obtained simultaneously. Thus, when the first network communication is normal, the first server can obtain the operation update data of each sub-industrial Internet of Things device through the second server, which is convenient for subsequent targeted management and maintenance of each sub-industrial Internet of Things device.

[0044] Step S220, store the operation update data through a distributed message service queue system to update the pre-stored historical operation data corresponding to the target device based on the operation update data.

[0045] A distributed message service queue system is a technical architecture used for message passing and communication in a distributed system. It has high reliability and scalability and can effectively store and process a large amount of data, such as data like text, binary data, or objects.

[0046] To improve the fault tolerance of the first server and ensure the storage security of the operation data, the distributed consumption service queue system deployed in the first server is used to store the operation data of the target device, so that other servers or application programs can obtain this data in a timely manner later, ensuring the storage reliability and processing efficiency of the operation data. Among them, the distributed consumption service queue system may include, but is not limited to: Kafka (Kafka Streams, Kafka) system, Rabbit Message Queue (RabbitMQ), Apache Active Message Queue (ActiveMQ), and can be specifically selected according to actual needs.

[0047] After obtaining the operation update data, the first server stores the obtained operation update data into the distributed message service queue system to use the operation update data to update the pre-stored historical operation data corresponding to the target device, so that the operation data corresponding to the target device stored in the first server can be updated in a timely manner and can correspond to the actual operation situation of the target device, thereby helping to ensure the accuracy and integrity of the data. Among them, the historical operation data refers to the operation data during the operation of the target device before the operation update data is obtained.

[0048] Step S230, in response to the communication status of the first network returning to normal, determine the unprocessed information to be consumed in the distributed message service queue system during the disconnection of the first network.

[0049] In response to the communication status of the first network returning to normal, it indicates that during the communication between the first server and the second server, the first network has experienced an interruption or other abnormal conditions, which will affect the integrity of the data transmission between the first server and the second server. Therefore, when it is detected that the communication status of the first network returns to normal, the first server will detect the distributed message service queue system to determine which information to be consumed has not been processed during the disconnection of the first network, so as to timely discover and process the information to be consumed accumulated during the network disconnection, avoiding data loss or delayed processing, and thus helping to ensure the performance of the first server. Among them, the information to be consumed is the information to be accessed and interacted with the operation data in the distributed message service queue system, and the operation data is the data obtained after the last update before the communication status of the first network is disconnected.

[0050] Step S240, based on the information to be consumed, access the operation data.

[0051] After determining the information to be consumed, the target operation to be performed on the operation data can be determined according to the information to be consumed, and the operation data can be accessed and processed according to the target operation, which can ensure that after the first network is restored, the accumulated information to be consumed can be processed in a timely manner, thereby helping to ensure the normal operation of the first server and improving the accuracy of data management. Among them, the target operation may include, but is not limited to: operations such as reading, writing, updating, merging, deleting, and restoring.

[0052] According to the data management method provided by the present invention, a connection between a first server and a second server is established in advance through a first network, and a connection between the second server and a target device is established through a second network, and the stability of the second network is higher than that of the first network, which can ensure that the operation data generated during the operation of the target device can be completely obtained by the second server. Furthermore, when the communication status of the first network is normal, the first server can timely obtain the operation update data of the target device from the second server and store it through a distributed message service queue system, so as to update the previously stored historical operation data corresponding to the target device based on the operation update data, so that the operation data corresponding to the target device stored in the first server can be updated in a timely manner and can correspond to the actual operation situation of the target device, which helps to ensure the accuracy and integrity of the data. And when it is detected that the communication status of the first network returns to normal, by determining the information to be consumed that has not been processed by the distributed message service queue system during the disconnection of the first network, it can be determined which information needs to be processed in a timely manner, and then access the operation data based on the information to be consumed, which can ensure the timeliness of processing the information to be consumed, ensure the integrity and reliability of the operation data, and thus help to improve the performance of the first server and facilitate subsequent targeted management and maintenance of the target device.

[0053] In some optional embodiments, the above step S210 may include the following steps:

[0054] Step a1, in response to the normal communication status of the first network, receive the operation update data of the target device sent by the second server.

[0055] Among them, the second server is determined based on the detection result of the data operation log of the target database. The target database is deployed in the second server and corresponds to the target device. The target database is used to record and save the current operation data of the target device. The data operation log records various operations on the database, including insert, update, and delete, etc.

[0056] When the communication status of the first network is normal, the second server can analyze the data update situation in the target database by detecting the data operation logs of the target database, so as to obtain a detection result, which provides a basis for determining the running updated data subsequently. Among them, the detection result can be characterized as that the data in the target database has not been updated, or it can be characterized as that the data in the target database has been updated.

[0057] If the detection result is characterized as that the data in the target database has been updated, the updated data will be used as the running updated data and sent to the first server, so that the first server can realize the real-time monitoring and acquisition of the running data of the target device, timely discover the data update, and achieve the purpose of data synchronization.

[0058] In one example, the data operation logs of the target database can be detected through a data capture tool, and then the detection result can be obtained. For example, if the target database is a MySQL database, the data change capture tool can be the Debezium plugin pre-deployed on the MySQL server. The second server can detect the data update situation of the data operation logs of the target database (binlog logs) by listening to the data operation logs of the target database through the Debezium plugin, so as to determine and obtain the running updated data of the target device.

[0059] In some optional embodiments, as Figure 3 shown, the above step S220 may include the following steps S221 to S223:

[0060] Step S221, perform a first encryption process on the running updated data to obtain a first encryption result.

[0061] To ensure the security of data transmission, the running updated data is transmitted in an encrypted transmission manner to avoid the situation that the running data is maliciously stolen or tampered with during the transmission process. Therefore, after determining the running updated data, the running updated data is subjected to a first encryption process through a first encryption algorithm, and then the encrypted first encryption result is obtained.

[0062] Among them, the first encryption algorithm may include, but is not limited to, symmetric encryption algorithms or asymmetric encryption algorithms. If a symmetric encryption algorithm (e.g., AES) is used as the first encryption algorithm, then during the data transmission process, the same key can be used for encryption and decryption processes, which can ensure the security of data transmission while helping to simplify the encryption / decryption process and improve data processing efficiency. If an asymmetric encryption algorithm (e.g., RSA) is used as the first encryption algorithm, it helps to reduce the risk of key leakage and improve data transmission security. Moreover, when using an asymmetric encryption algorithm for encryption, a digital signature will be generated, which helps to ensure the integrity of data transmission and the authenticity of the source. The actual first encryption algorithm can be determined according to requirements and is not limited here.

[0063] Step S222, store the first encryption processing result through the distributed message service queue system, so that the distributed message service queue system updates the pre-stored historical operation data corresponding to the target device based on the first decryption processing result of the first encryption processing result to obtain an update result.

[0064] The distributed message service queue system has the capabilities of data storage and processing. Therefore, storing the first encryption processing result in the distributed message service queue system can ensure the storage reliability and update timeliness of the operation update data.

[0065] In the distributed message service queue system, the historical operation data corresponding to the target device is pre-stored. Among them, the historical operation data is the operation data that is continuously updated as the target device runs. When the historical operation data needs to be updated, the distributed message service queue system performs the first decryption processing on the first encryption processing result to obtain the decrypted operation update data, and then merges the operation update data with the corresponding historical operation data to obtain the update data corresponding to the running state of the target device, thereby realizing the secure update and storage of the operation data of the target device. Among them, the first decryption processing corresponding to the first decryption processing result corresponds to the first encryption processing.

[0066] In some optional examples, the distributed message service queue system may include multiple intermediate storage nodes to improve the fault tolerance of the distributed message service queue system. The process of storing the first encryption processing result through the distributed message service queue system may include: storing the first encryption processing result through multiple intermediate storage nodes respectively, which helps to improve the reliability and availability of data storage.

[0067] In some other alternative examples, the distributed message service queue system may have an intermediate storage node. Thus, when storing the first encrypted processing result through the distributed message service queue system, the first encrypted processing result can be stored in the intermediate storage node for centralized management and monitoring, reducing the complexity of the system and the maintenance cost.

[0068] Step S223: Cache the update result in the distributed message service queue system through the target search engine.

[0069] The target search engine is connected to the distributed message service queue system and can communicate and interact based on the running data in the distributed message service queue system. Among them, the target search engine can be a distributed search engine to meet the requirements of distributed search and data storage, providing high availability and fault tolerance. For example, the target search engine can be Elasticsearch or other systems capable of searching for information on the Internet.

[0070] Caching the update result through the target search engine can improve the ability to meet the demand for accessing the running data of the target device and reduce the repeated queries to the original data source, thus helping to improve the access speed and efficiency of the data.

[0071] In some alternative examples, the above step S223 includes:

[0072] Step b1: Perform a second encryption process on the update result through the distributed message service queue system to obtain a second encrypted result, and send the second encrypted result to the target search engine;

[0073] Step b2: Perform a second decryption process on the second encrypted result through the target search engine to cache the update result according to the obtained second decryption result.

[0074] Specifically, to ensure the security of data transmission, the distributed message service queue system and the target search engine use the form of encryption with the second encryption algorithm during the data transmission process. Among them, the second encryption algorithm can be the same as the first encryption algorithm or different from the first encryption algorithm. If the same first encryption algorithm and second encryption algorithm are used, the complexity of implementation can be reduced, which is beneficial to improving the efficiency of encryption and decryption. If different first encryption algorithms and second encryption algorithms are used, the security of data transmission can be increased, which is beneficial to meeting the usage requirements of multiple scenarios and reducing the risk of data leakage or tampering.

[0075] The updated result is secondarily encrypted through a distributed message service queue system to obtain a second encrypted result, so as to further enhance the data security. Then, the second encrypted result is sent to the target search engine. After receiving the second encrypted result, the target search engine performs second decryption processing on it and caches the updated result according to the obtained second decryption result for subsequent fast access and use, thereby reducing repeated queries on the original data source and improving the data access speed and efficiency. Among them, the second decryption processing corresponds to the second encrypted result.

[0076] By storing and running the updated data in the above manner, the security and confidentiality of the updated result can be ensured, and at the same time, the data access speed and efficiency can be improved. And during the transmission process, through encryption processing, the updated result can be prevented from being illegally obtained and tampered with, improving the data security, reliability and access efficiency, thus helping to meet various data management requirements.

[0077] In some optional embodiments, the above step S230 includes the following steps:

[0078] Step c1, in response to the communication status of the first network returning to normal, the target search engine periodically creates a snapshot of the distributed message service queue system to determine the unprocessed information to be consumed in the distributed message service queue system during the interruption of the first network.

[0079] The distributed message service queue system has a message persistence mechanism. Whether the network is abnormal or not, the information to be consumed sent into the distributed message service queue system or the information to be consumed that the distributed message service queue system needs to send out will not be lost.

[0080] To ensure the performance of the distributed message service queue system, the snapshot function provided by the target search engine is used to periodically create a snapshot of the distributed message service queue system, so that when the communication status of the first network is abnormal, the unprocessed information to be consumed accumulated during the network interruption can be discovered and processed in time, ensuring the normal operation of the system and the accuracy of the data. In one example, the period for creating a snapshot of the distributed message service queue system can be created in seconds, which helps to ensure the effectiveness and timeliness of the determination of the information to be consumed.

[0081] Since the snapshot is created before the network interruption, it can provide information about the system state during the interruption. Therefore, when it is detected that the communication state of the first network returns to normal, the target search engine can be used to regularly analyze the snapshots created by the distributed message service queue system to determine which messages in the distributed message service queue system have not been processed or consumed during the first network interruption, and then determine the unprocessed information to be consumed during the first network interruption, which is convenient for timely processing to avoid data loss or delayed processing, thus helping to improve the reliability and stability of the distributed message service queue system.

[0082] Exemplary embodiments of the present disclosure provide a data management method applied to a first server, as Figure 4 shown, the data management method may include the following steps:

[0083] Step S310, in response to the normal communication state of the first network, obtain the running update data of the target device through the second server. For details, please refer to Figure 2 step S210 of the embodiment shown, which will not be elaborated here.

[0084] Step S320, store the running update data through the distributed message service queue system to update the pre-stored historical running data corresponding to the target device based on the running update data. For details, please refer to Figure 2 step S220 of the embodiment shown, which will not be elaborated here.

[0085] Step S330, in response to the communication state of the first network returning to normal, regularly determine the unprocessed information to be consumed in the distributed message service queue system during the interruption of the first network through the target search engine for the snapshots created by the distributed message service queue system.

[0086] Step S340, access the running data based on the information to be consumed. For details, please refer to Figure 2 step S240 of the embodiment shown, which will not be elaborated here.

[0087] Step S350, obtain the content update data of the target device from the file management server.

[0088] Among them, the content update data is obtained by the file management server from the second server. The file management server is connected to the first server and the second server through the first network. The content update data may include, but is not limited to, data such as media data and files related to the operation of the target device. Since the data format of the content update data is not formatted data, it may not be directly and effectively obtained from the second server during the data transmission process between the second server and the first server. Therefore, in order for the first server to obtain the complete data involved in the operation of the target device, after the second server obtains the content data during the operation of the target device, it transmits it to the file management server. The file management server identifies and extracts the content update data in the content data, and then the first server obtains the content update data of the target device through the file management server to ensure that the first server can obtain the latest content update data of the target device in a timely manner and maintain the synchronization and consistency of the data.

[0089] Step S360, caching the content update data through the target search engine for the client of the target search engine to access.

[0090] The first server sends the obtained content update data to the target search engine. After receiving the content update data, the target search engine caches it for the client to access. The client can obtain the latest content update data by accessing the target search engine, improving the data access efficiency and response speed, which helps to improve the data availability and access speed and facilitates the client to obtain the latest information in a timely manner.

[0091] According to the data management method provided by the present disclosure, by obtaining the content update data of the target device from the file management server and caching it through the target search engine, the timely update and efficient access of the data are realized, so that a better service experience can be provided for visitors.

[0092] In some optional embodiments, the operation data can be displayed through a data visualization interface, and then the operation data of the target device can be monitored and analyzed in real time according to requirements, which is convenient for subsequent targeted management and maintenance of the operation of the target device.

[0093] Based on the same inventive concept, the present disclosure also provides a data management system. As Figure 5 shown, the data management system 100 includes a target device 110, a second server 120, and a first server 130.

[0094] The second server 120 is connected to the target device 110 through the second network and is used to record and save the current operation data of the target device 110;

[0095] The first server 130 is connected to the second server 120 via a first network. The stability of the second network is higher than that of the first network. The first server 130 is used to obtain the operation update data of the target device 110 through the second server 120, store the operation update data through the distributed message service queue system 131, so as to update the pre-stored historical operation data corresponding to the target device 110 based on the operation update data, determine the unprocessed information to be consumed in the distributed message service queue system 131 during the disconnection of the first network, and access the operation data based on the information to be consumed. Among them, the operation update data is determined based on the current operation data, the information to be consumed is the information to be accessed and interacted with the operation data in the distributed message service queue system 131, and the operation data is the data obtained after the last update of the distributed message service queue system 131 before the communication state of the first network is disconnected.

[0096] Among them, the specific implementation process of the first server 130 managing the operation data of the target device 110 based on the connection with the second server 120 can be implemented by using the embodiments of any of the above data management methods provided by the present invention, and will not be elaborated in detail here.

[0097] In some optional embodiments, the distributed message service queue system 131 includes multiple intermediate storage nodes, which are respectively deployed on corresponding physical servers, so that when a certain physical server fails, the intermediate storage nodes on other physical servers can still continue to work, ensuring the reliability and thus fault tolerance of data transmission, which is beneficial to improving the performance of the distributed message service queue system 131.

[0098] In some optional embodiments, as Figure 5 shown, the first server 130 further includes a target search engine 132, and the target search engine 132 is used to cache the operation data in the distributed message service queue system, so that the client 200 of the target search engine 132 can access the operation data of the target device 110 through the target search engine 132 according to the needs.

[0099] In some optional embodiments, as Figure 6 shown, the data management system 100 further includes: a file management server 140, which is connected to the second server 120 and the first server 130 respectively through the first network. The file management server 140 is used to obtain the content update data of the target device 110 from the second server 120, and send the content update data to the target search engine 132, so as to cache the content update data through the target search engine 132 for the client 200 of the target search engine 132 to access.

[0100] In some optionally implemented scenarios, taking the target device as an ore sorter as an example, the process of constructing a data management system can be as follows:

[0101] Previously, connect the main body of the ore sorter and multiple sub-industrial Internet devices to the second server through the second network (intranet), and use the Modbus TCP protocol to enable the second server to exchange data with the ore sorter and the corresponding multiple sub-industrial Internet devices, so that the second server can record and save the operation data of the ore sorter.

[0102] Deploy the change capture tool (Debezium plugin) of the distributed message service queue system (kafka) on the MySQL server, so that when the second server updates the data in the target database in MySQL format, it can detect the data update situation of the binlog log according to the data operation log (binlog log) of the target database, and then determine the operation update data of the target device.

[0103] Construct a distributed message service queue system (kafka cluster) composed of three intermediate storage nodes (brokers), and different intermediate storage nodes are deployed on different physical servers. Establish a connection between the Debezium plugin and each intermediate storage node, so that during the operation of the target device, it can be used as a producer to transmit data with the kafka cluster, enabling the kafka cluster to obtain the operation update data of the target device in a timely manner.

[0104] Integrate the kafka cluster and the target search engine on the first server for unified management by the first server. Among them, the client of the target search engine can be a consumer of the kafka cluster, and the access to the target search engine can be realized through a consumer application program constructed by SpringCloud (an ordered set of a series of frameworks). The number of target search engines can be at least one.

[0105] Establish a communication connection between the first server and the second server through the first network (extranet), and then combine with the ore sorter connected to the second server to form the required data management system.

[0106] The process of managing the data of the ore sorter through the above data management system can be as follows:

[0107] According to the communication connection between the ore separator and the second server, the second server records the operation data of the ore separator and stores it in the target database corresponding to the ore separator internally. Inside the second server, the Debezium plug-in is used to detect data updates in the binlog of the target database, determine the operation update data of the ore separator, and send the operation update data to the kafka cluster integrated in the first server when the first network communication status is normal.

[0108] The historical operation data of the ore separator during operation is stored in the kafka cluster. Furthermore, according to the operation update data of the ore separator transmitted by the Debezium plug-in, the historical operation data of the ore separator can be updated in a timely manner, thus ensuring the timeliness of data update. In one example, during the data transmission process between the Debezium plug-in and the kafka cluster, encryption transmission can be performed through the first encryption algorithm.

[0109] The kafka cluster sends the updated operation data of the ore separator to the target search engine for caching by the target search engine. If the number of target search engines is one, the kafka cluster can directly send the updated operation data of the ore separator to the target search engine. If the number of target search engines is multiple, the kafka cluster can send the updated operation data to each target search engine in a broadcast form. In one example, during the data transmission process between the kafka cluster and the target search engine, encryption transmission can be performed through the second encryption algorithm.

[0110] When the first network communication status is normal, the target search engine can periodically create a snapshot of the kafka cluster to clarify the information consumption and processing situation of the kafka cluster.

[0111] In some examples, when the first network communication status is normal, the first server can obtain the content update data of the target device through the file management server. Among them, the content update data is obtained by the file management server from the second server, and the file management server is connected to the first server and the second server through the first network.

[0112] When the first network is interrupted and then resumes normal operation, the snapshot created by the kafka cluster can be used to determine the unprocessed information to be consumed by the kafka cluster during the interruption of the first network, and then targeted processing can be performed through the kafka cluster to ensure the performance of the data management system.

[0113] To enable the client of the target search engine to intuitively and quickly view the operation data of the ore sorter, data can be displayed through a data visualization interface, so that even in the case of a first network interruption, the historical operation data of the ore sorter can also be displayed, which helps to timely understand the operation of the ore sorter and facilitates its targeted management.

[0114] Through the data management system and the corresponding data management method provided by the present disclosure, it can provide comprehensive support for efficient, secure, and intelligent management of modern industrial data transmission, thereby providing a more reliable and efficient solution for industrial data transmission and helping to improve the intelligent level of the industrial production process.

[0115] Based on the same inventive concept, as Figure 7 shown, the present disclosure also provides a data management device 400, which is applied to a first server and includes:

[0116] An acquisition module 410, configured to obtain the operation update data of the target device through a second server in response to the normal communication state of the first network. The second server is connected to the target device through a second network and is used to record and save the current operation data of the target device. The operation update data is determined based on the current operation data, and the stability of the second network is higher than that of the first network;

[0117] A storage module 420, configured to store the operation update data through a distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data;

[0118] A first processing module 430, configured to determine the unprocessed information to be consumed in the distributed message service queue system during the disconnection of the first network in response to the recovery of the normal communication state of the first network. The information to be consumed is the information to be accessed and interacted with the operation data in the distributed message service queue system, and the operation data is the data obtained after the last update of the distributed message service queue system before the communication state of the first network is disconnected;

[0119] A second processing module 440, configured to access the operation data based on the information to be consumed.

[0120] In some optional embodiments, the first acquisition module 410 includes: a receiving unit, configured to receive the operation update data of the target device sent by the second server, where the second server is determined based on the detection result of the data operation log of the target database. The target database is deployed in the second server and corresponds to the target device, and the target database is used to record and save the current operation data of the target device.

[0121] In some optional embodiments, the storage module 420 includes: a first processing unit configured to perform a first encryption process on the running update data to obtain a first encryption result; a second processing unit configured to store the first encryption result through a distributed message service queue system, so that the distributed message service queue system updates the pre-stored historical running data corresponding to the target device based on a first decryption result of the first encryption result to obtain an update result, wherein the first decryption corresponding to the first decryption result corresponds to the first encryption process; a third processing unit configured to cache the update result in the distributed message service queue system through a target search engine.

[0122] In some optional embodiments, the distributed message service queue system includes a plurality of intermediate storage nodes; the second processing unit includes: a storage unit configured to store the first encryption result through the plurality of intermediate storage nodes respectively.

[0123] In some optional embodiments, the third processing unit includes: a first execution unit configured to perform a second encryption process on the update result through the distributed message service queue system to obtain a second encryption result, and send the second encryption result to the target search engine; a second execution unit configured to perform a second decryption process on the second encryption result through the target search engine to cache the update result according to the obtained second decryption result, wherein the second decryption process corresponds to the second encryption result.

[0124] In some optional embodiments, the first processing module 430 includes: a fourth processing unit configured to, in response to the communication state of the first network returning to normal, periodically create a snapshot of the distributed message service queue system through the target search engine to determine the unprocessed information to be consumed by the distributed message service queue system during the interruption of the first network.

[0125] In some optional embodiments, the apparatus further includes: a second acquisition module configured to acquire content update data of the target device from a file management server, wherein the content update data is acquired by the file management server from a second server, and the file management server is connected to a first server and a second server through a first network; a third processing module configured to cache the content update data through the target search engine for access by a client of the target search engine.

[0126] In some optional embodiments, the apparatus further includes: a display unit configured to display running data through a data visualization interface.

[0127] Regarding the data management apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0128] Based on the same inventive concept, the present disclosure also provides a computer-readable storage medium storing the following program for executing the image segmentation method of any of the foregoing embodiments.

[0129] The present application uses specific terms to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that the "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more at different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the present application can be appropriately combined.

[0130] In the context of the present application, unless the context clearly indicates an exception, the words "a", "an", "one", and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0131] Similarly, it should be noted that, in order to simplify the description of the present application and thus help the understanding of one or more embodiments of the application, in the foregoing description of the embodiments of the present application, sometimes multiple features are merged into one embodiment, drawing, or description thereof. However, this disclosure method does not mean that the features required by the subject matter of the present application are more than those mentioned in the claims. In fact, the features of the embodiment are less than all the features of the single embodiment disclosed above.

[0132] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is only an example and does not constitute a limitation to the present application. Although not explicitly stated here, those skilled in the art may make various modifications, improvements, and corrections to the present application. Such modifications, improvements, and corrections are proposed in the present application, so such modifications, improvements, and corrections still fall within the spirit and scope of the embodiments of the present application.

Claims

1. A data management method, applied to a first server, comprising: In response to the communication status of the first network being normal, obtaining operation update data of the target device through a second server, wherein the second server is connected to the target device through a second network and is used to record and save current operation data of the target device, the operation update data is determined based on the current operation data, and the stability of the second network is higher than the stability of the first network; and storing the operation update data through a distributed message service queue system to update the pre-stored historical operation data corresponding to the target device based on the operation update data; In response to the communication state of the first network returning to normal, determining the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network, including: in response to the communication state of the first network returning to normal, periodically creating snapshots of the distributed message service queue system through a target search engine, and determining the unprocessed information to be consumed by the distributed message service queue system during the interruption of the first network, the information to be consumed being information to be accessed and interacted with the operating data in the distributed message service queue system, the operating data being data obtained after the distributed message service queue system was last updated before the communication state of the first network was disconnected; and, Based on the information to be consumed, the operation data is accessed.

2. The data management method according to claim 1, wherein: The step of acquiring the operation update data of the target device through the second server in response to the communication state of the first network being normal includes: In response to the communication status of the first network being normal, receiving the operation update data of the target device sent by the second server, wherein the second server is determined based on the detection result of the data operation log of the target database, the target database is deployed in the second server and corresponds to the target device, and the target database is used to record and save the current operation data of the target device.

3. The data management method according to claim 2, wherein: The storing the operation update data through the distributed message service queue system to update the pre-stored historical operation data corresponding to the target device based on the operation update data includes: Performing a first encryption process on the operation update data to obtain a first encryption process result; The first encryption processing result is stored by the distributed message service queue system, so that the distributed message service queue system updates the pre-stored historical operation data corresponding to the target device based on the first decryption processing result of the first encryption processing result to obtain an update result, wherein the first decryption processing corresponding to the first decryption processing result corresponds to the first encryption processing; The update result in the distributed message service queue system is cached by the target search engine.

4. The data management method according to claim 3, wherein: The distributed message service queue system includes a plurality of intermediate storage nodes; The storing the first encryption processing result by the distributed message service queue system includes: The first encryption processing result is stored respectively by a plurality of intermediate storage nodes.

5. The data management method according to claim 3 or 4, wherein: The step of caching the update result in the distributed message service queue system through a target search engine includes: Performing a second encryption process on the update result through the distributed message service queue system to obtain a second encryption result, and sending the second encryption result to the target search engine; The second encryption result is subjected to a second decryption process by the target search engine, so as to cache the update result according to the obtained second decryption result, wherein the second decryption process corresponds to the second encryption result.

6. The data management method according to claim 1, wherein: The method further comprises: Acquire content update data of the target device from a file management server, wherein the content update data is acquired by the file management server from the second server, and the file management server is connected to the first server and the second server through the first network; The content update data is cached by the target search engine for access by a client of the target search engine.

7. The data management method according to claim 1, wherein: The method further comprises: The operating data is displayed through a data visualization interface.

8. A data management device, applied to a first server, comprising: a first acquisition module, configured to acquire, in response to a normal communication state of the first network, operation update data of the target device through a second server, wherein the second server is connected to the target device through a second network, and is configured to record and save current operation data of the target device, the operation update data is determined based on the current operation data, and the stability of the second network is higher than the stability of the first network; A storage module, configured to store the operation update data through a distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data; A first processing module is used to determine, in response to the communication state of the first network returning to normal, information to be consumed that has not been processed by the distributed message service queue system during the disconnection of the first network, including: in response to the communication state of the first network returning to normal, regularly creating snapshots of the distributed message service queue system through a target search engine, and determining information to be consumed that has not been processed by the distributed message service queue system during the interruption of the first network, the information to be consumed being information to be accessed and interacted with operating data in the distributed message service queue system, the operating data being data obtained after the distributed message service queue system was last updated before the communication state of the first network was disconnected; The second processing module is used to access the operation data based on the information to be consumed.

9. A data management system comprising: Target device; A second server, connected to the target device via a second network, and configured to record and save current operation data of the target device; A first server is connected to the second server through a first network, the stability of the second network is higher than that of the first network, the first server is used to obtain the operation update data of the target device through the second server, store the operation update data through a distributed message service queue system, and update the pre-stored historical operation data corresponding to the target device based on the operation update data, and in response to the communication state of the first network returning to normal, determine the unprocessed to-be-consumed information of the distributed message service queue system during the disconnection of the first network, including: in response to the communication state of the first network returning to normal, periodically create a snapshot of the distributed message service queue system through a target search engine, determine the unprocessed to-be-consumed information of the distributed message service queue system during the interruption of the first network, and access the operation data based on the to-be-consumed information, wherein the operation update data is determined based on the current operation data, the to-be-consumed information is information to be accessed and interacted with the operation data in the distributed message service queue system, and the operation data is data obtained after the distributed message service queue system is last updated before the communication state of the first network is disconnected.

10. The data management system according to claim 9, wherein: The distributed message service queue system includes a plurality of intermediate storage nodes, which are respectively deployed on corresponding physical servers.

11. The data management system according to claim 9 or 10, wherein: The first server includes a target search engine, and the target search engine is used to cache the operating data in the distributed message service queue system.

12. The data management system according to claim 11, wherein: The data management system also includes: a file management server, which is connected to the second server and the first server respectively through the first network, and the file management server is used to obtain content update data of the target device from the second server, and send the content update data to the target search engine, so as to cache the content update data through the target search engine for access by the client of the target search engine.

13. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data management method according to any one of claims 1 to 7.

Images