Data management method, device and system and readable storage medium

Abstract

The invention relates to the technical field of computers, in particular to a data management method, device and system and a readable storage medium. The data management method is applied to a first server, and comprises the following steps: in response to a normal communication state of a first network, obtaining operation update data of target equipment through a second server; and storing the operation update data through the distributed message service queue system, so as to update pre-stored historical operation data corresponding to the target equipment based on the operation update data, in response to recovery of the communication state of the first network to normal, determining unprocessed to-be-consumed information of the distributed message service queue system in the disconnection process of the first network; and accessing the operation data based on the to-be-consumed information. According to the embodiment of the invention, the acquisition accuracy and integrity of the operation data can be guaranteed, and when the communication state of the first network returns to normal, the processing timeliness of the to-be-consumed information can also be guaranteed, thereby facilitating the improvement of the performance, and facilitating the subsequent targeted management and maintenance of the target equipment.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data management method, a data management method, a data management device, a data management system and a computer-readable storage medium. Background Art

[0002] With the continuous development of industrial automation, the real-time and reliability of data transmission have become important indicators for measuring system performance. In related technologies, data information of devices is directly obtained from devices through network transmission, but this data transmission method relies on a single data storage and transmission path. When the network is abnormal during data transmission, it will affect the integrity of data information acquisition, thereby limiting the level of intelligence in the industrial production process. Summary of the invention

[0003] To overcome the problems existing in the related art, an exemplary embodiment of the present disclosure discloses a data management method, which is applied to a first server, including: in response to the communication status of the first network being normal, obtaining operation update data of a target device through a second server, wherein the second server is connected to the target device through a second network, and is used to record and save the current operation data of the target device, and the operation update data is determined based on the current operation data, and the stability of the second network is higher than the stability of the first network; and, storing the operation update data through a distributed message service queue system to update the pre-stored historical operation data corresponding to the target device based on the operation update data; in response to the communication status of the first network returning to normal, determining the information to be consumed that has not been processed by the distributed message service queue system during the disconnection of the first network, the information to be consumed is information to be accessed and interacted with the operation data in the distributed message service queue system, and the operation data is data obtained after the distributed message service queue system was last updated before the communication status of the first network was disconnected; and, based on the information to be consumed, accessing the operation data.

[0004] In some embodiments, in response to the communication status of the first network being normal, obtaining the operation update data of the target device through the second server includes: receiving the operation update data of the target device sent by the second server, wherein the second server is determined based on the detection result of the data operation log of the target database, the target database is deployed in the second server and corresponds to the target device, and the target database is used to record and save the current operation data of the target device.

[0005] In some embodiments, operation update data is stored through a distributed message service queue system to update pre-stored historical operation data corresponding to a target device based on the operation update data, including: performing a first encryption processing on the operation update data to obtain a first encryption processing result; storing the first encryption processing result through a distributed message service queue system so that the distributed message service queue system updates the pre-stored historical operation data corresponding to the target device based on a first decryption processing result of the first encryption processing result to obtain an update result, wherein the first decryption processing corresponding to the first decryption processing result corresponds to the first encryption processing; caching the update result in the distributed message service queue system through the target search engine.

[0006] In some embodiments, the distributed message service queue system includes multiple intermediate storage nodes; storing the first encryption processing result through the distributed message service queue system includes: storing the first encryption processing result through the multiple intermediate storage nodes respectively.

[0007] In some embodiments, caching update results in a distributed message service queue system includes: performing a second encryption process on the update results through the distributed message service queue system to obtain a second encryption result, and sending the second encryption result to a target search engine; performing a second decryption process on the second encryption result through the target search engine to cache the update results based on the obtained second decryption result, wherein the second decryption process corresponds to the second encryption result.

[0008] In some embodiments, in response to the communication status of the first network returning to normal, determining the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network includes: in response to the communication status of the first network returning to normal, regularly creating snapshots of the distributed message service queue system through the target search engine, and determining the unprocessed information to be consumed by the distributed message service queue system during the interruption of the first network.

[0009] In some embodiments, the method further includes: obtaining content update data of the target device from a file management server, wherein the content update data is obtained by the file management server from a second server, and the file management server is connected to the first server and the second server via a first network; caching the content update data via a target search engine for access by a client of the target search engine.

[0010] In some embodiments, the method further includes: displaying the operating data through a data visualization interface.

[0011] In a second aspect, the present disclosure also provides a data management device, which is applied to a first server, including: an acquisition module, which is used to obtain operation update data of a target device through a second server in response to the communication status of the first network being normal, wherein the second server is connected to the target device through a second network, and is used to record and save the current operation data of the target device, and the operation update data is determined based on the current operation data, and the stability of the second network is higher than the stability of the first network; a storage module, which is used to store the operation update data through a distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data; a first processing module, which is used to determine, in response to the communication status of the first network returning to normal, the information to be consumed that has not been processed by the distributed message service queue system during the disconnection of the first network, the information to be consumed is information to be accessed and interacted with the operation data in the distributed message service queue system, and the operation data is data obtained after the distributed message service queue system was last updated before the communication status of the first network was disconnected; a second processing module, which is used to access the operation data based on the information to be consumed.

[0012] In a third aspect, the present disclosure also provides a data management system, including: a target device; a second server, connected to the target device via a second network, and used to record and save the current operation data of the target device; a first server, connected to the second server via a first network, the stability of the second network is higher than the stability of the first network, the first server is used to obtain the operation update data of the target device through the second server, store the operation update data through a distributed message service queue system, update the pre-stored historical operation data corresponding to the target device based on the operation update data, determine the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network, and access the operation data based on the information to be consumed, wherein the operation update data is determined based on the current operation data, the information to be consumed is information to be accessed and interacted with the operation data in the distributed message service queue system, and the operation data is data obtained after the distributed message service queue system was last updated before the communication state of the first network was disconnected.

[0013] In some embodiments, the distributed message service queue system includes multiple intermediate storage nodes, which are respectively deployed on corresponding physical servers.

[0014] In some embodiments, the first server includes a target search engine, and the target search engine is used to cache operating data in the distributed message service queue system.

[0015] In some embodiments, the data management system also includes: a file management server, which is connected to the second server and the first server respectively through the first network, and the file management server is used to obtain content update data of the target device from the second server, and send the content update data to the target search engine, so as to cache the content update data through the target search engine for access by the client of the target search engine.

[0016] In a fourth aspect, the present disclosure further provides a computer-readable storage medium, which stores the following program, and the program is used to execute the data management method as any of the above embodiments.

[0017] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: According to the data management method provided by the present disclosure, a connection is established between the first server and the second server through the first network in advance, and a connection is established between the second server and the target device through the second network, and the stability of the second network is higher than that of the first network, which can ensure that the operation data generated by the target device during operation can be completely acquired by the second server, and then, when the communication state of the first network is normal, the first server can timely acquire the operation update data of the target device from the second server, and store it through the distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data, so that the operation data corresponding to the target device stored in the first server can be updated in time, can correspond to the actual operation status of the target device, and help to ensure the accuracy and integrity of the data. In addition, when it is detected that the communication state of the first network has returned to normal, by determining the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network, it can be clarified which information needs to be processed in time, and then the operation data is accessed based on the information to be consumed, which can ensure the timeliness of processing the information to be consumed, and ensure the integrity and reliability of the operation data, thereby helping to improve the performance of the first server and facilitate the subsequent targeted management and maintenance of the target device.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention may be better understood by describing exemplary embodiments of the present invention in conjunction with the accompanying drawings, in which:

[0020] Figure 1 is a schematic diagram of the architecture of a data management system according to an exemplary embodiment of the disclosure;

[0021] Figure 2 It is a flowchart of a data management method according to an exemplary embodiment of the disclosure;

[0022] Figure 3 is a flowchart of another data management method according to an exemplary embodiment of the disclosure;

[0023] Figure 4 is a flowchart of another data management method according to an exemplary embodiment of the disclosure;

[0024] Figure 5 is a schematic diagram of another data management system according to an exemplary embodiment of the disclosure;

[0025] Figure 6 is a schematic diagram of the architecture of another data management system according to an exemplary embodiment of the disclosure;

[0026] Figure 7 The present invention is a schematic diagram of a data management device according to an exemplary embodiment of the present invention. Specific embodiments

[0027] The specific embodiments of the present invention will be described below. It should be noted that in the specific description of these embodiments, in order to be concise and clear, this specification cannot provide a detailed description of all the features of the actual embodiments. It should be understood that in the actual implementation of any embodiment, just as in the process of any engineering project or design project, in order to achieve the specific goals of the developer and to meet system-related or business-related restrictions, various specific decisions are often made, and this will also change from one embodiment to another. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the content disclosed by the present invention, some changes such as design, manufacturing or production based on the technical content disclosed in the present disclosure are just conventional technical means, and should not be understood as insufficient content of the present disclosure.

[0028] Unless otherwise defined, the technical or scientific terms used in the claims and the specification shall have the usual meaning understood by persons with ordinary skills in the technical field to which the invention belongs. The words "first", "second" and similar words used in the patent application specification and the claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "One" or "one" and other similar words do not indicate a quantitative limitation, but indicate the existence of at least one. "Include" or "comprises" and other similar words mean that the elements or objects appearing before "include" or "comprises" include the elements or objects listed after "include" or "comprises" and their equivalent elements, and do not exclude other elements or objects. "Connected" or "connected" and other similar words are not limited to physical or mechanical connections, nor are they limited to direct or indirect connections.

[0029] In the related technology, the operation data of the equipment is directly obtained from the equipment through network transmission. That is, the industrial Internet equipment is pre-connected with the equipment server deployed in the same network environment, and then based on the communication connection between the two, the equipment server obtains and stores the data information of the industrial Internet equipment.

[0030] This data transmission method relies on a single data storage and transmission path. When an abnormality occurs in the network connecting the industrial Internet device and the device server during the data transmission process (for example, network interruption, congestion, delay, etc.), it will affect the integrity of the device server's acquisition of data information, thereby limiting the level of intelligence in the industrial production process.

[0031] To solve the above problems, the exemplary embodiments of the present disclosure provide a data management system, such as Figure 1 As shown, the data management system 100 includes: a target device 110 , a second server 120 and a first server 130 .

[0032] The target device 110 may be any industrial Internet device that can communicate over the network. For example, the target device 110 may include but is not limited to an intelligent ore dressing machine, a CNC machine tool, an industrial robot, etc., and may be specifically determined according to actual data management requirements.

[0033] The second server 120 is a server that can communicate and connect with the target device 110 and the first server 130 under different network environments. The second server 120 can obtain and store the operation data of the target device 110 based on the communication connection with the target device 110, and provide the operation data of the target device 110 to the first server 130 based on the communication connection with the first server 130, so that the first server 130 can subsequently perform targeted maintenance on the target device 110 based on the obtained operation data. The second server 120 can be a highly available server, and thus in the process of connecting with the target device 110 and the first server 130, even if a fault or abnormality occurs, it can continue to provide services. For example, the second server 120 can be a cluster of database servers (Cluster of Database Servers, CDS), a co-location server, a backup server, etc. Among them, CDS is a cluster system formed by combining multiple database servers together, which can improve the availability, reliability and performance of data by distributing data on multiple servers.

[0034] In some examples, the second server 120 can be a CDS, and each server in the CDS can obtain and store the operating data of the target device 110, and provide consistent services by sharing data and coordinating work. When the current server fails or fails, other servers can take over its work to ensure the continuity of the service, thereby effectively reducing the risk of service interruption, effectively ensuring the stability of data transmission between the target device 110 and the CDS, and ensuring the integrity of the acquisition of operating data.

[0035] In other examples, the second server 120 and the target device 110 may be connected via a wired connection to perform data transmission, thereby ensuring the stability and reliability of data transmission and reducing the risk of data loss and interference. At the same time, the wired connection can also provide higher transmission speed and lower latency, which is suitable for scenarios with higher requirements for data transmission.

[0036] The first server 130 may be a device server for storing and managing the operation data of the target device 110. The first server 130 may include, but is not limited to, the following features: having multiple interfaces, supporting multiple communication protocols, having certain data processing capabilities, and being able to ensure data transmission security. The multiple interfaces may include: Ethernet, serial port, USB, etc., so that different types of target devices 110 or second servers 120 can be connected according to requirements. The types of communication protocols may include, but are not limited to: Transmission Control Protocol / Internet Protocol (TCP / IP), Modbus TCP (an industrial automation communication protocol based on TCP / IP protocol), UDP (a connectionless network protocol), HTTP (an application layer protocol for the World Wide Web), etc., so that different types of target devices 110 or second servers 120 can be communicated with.

[0037] In some optional implementation scenarios, the second server 120 is connected to the first server 130 via a first network, and the target device 110 is connected to the second server 120 via a second network. By using different network environments, controlling the second server 120 to establish connections with the first server 130 and the target device 110 respectively helps to enhance the reliability of the network environment, facilitates network environment isolation, and ensures the transmission security of the operating data of the target device 110, thereby helping to meet various data management requirements.

[0038] Based on the same inventive concept, an exemplary embodiment of the present disclosure provides a data management method, which is applied to the first server in the above data management system. Figure 2 As shown, the data management method may include the following steps:

[0039] Step S210: In response to the communication status of the first network being normal, obtaining operation update data of the target device through the second server.

[0040] The first network is a network for the first server to communicate with the second server. If the communication status of the first network is normal, it indicates that the communication status between the first server and the second server is in a normal state. The setting location of the target device depends on its actual application requirements. Therefore, in order to ensure the stability of the second server obtaining the current operating data of the target device, the second server is controlled to be connected to the target device through the second network to record and save the current operating data of the target device. By using different networks to control the connection between the second server and the first server and the target device, it is helpful to enhance the reliability of the network environment, to achieve network environment isolation, and to ensure the transmission security of the operating data of the target device. Among them, the stability of the second network is higher than that of the first network, and then when data is transmitted in the network environment provided by the second network, the integrity of the current operating data obtained by the second server can be guaranteed.

[0041] Since the second server is a server for recording and saving the current operation data of the target device, the second server includes complete data information of the target device during operation. In response to the normal communication status of the first network, it is characterized that the first server and the second server can communicate normally, but since the operation data generated by the target device will be constantly updated and changed during operation, in order to improve data transmission efficiency and reduce redundant data, the current operation data of the target device obtained by the second server is matched with the existing historical operation data, and the new operation update data generated by the target device during operation is determined, and the operation update data is obtained. That is, when the first network is normal, the first server can obtain the latest operation data of the target device in a timely manner based on the connection with the second server, so that the first server can monitor and manage the operation status of the target device in real time, thereby helping to improve the reliability and stability of the data management system, facilitate timely discovery and resolution of problems, and ensure the normal operation of the target device. Among them, the operation update data may include but is not limited to the status information, performance indicators, configuration parameters and other data of the target device.

[0042] In some optional examples, the first network may be an external network, and the second network may be an intranet. The external network, also known as the Internet, refers to a computer network that is interconnected worldwide, with a large network range and high openness. By establishing a communication connection between the first server and the second server through the external network, the flexibility of the connection can be improved to meet various data management needs. The intranet, also known as a local area network, is a computer network that can be interconnected within an enterprise or organization, and has certain privacy and security. By controlling the connection between the target device and the second server through the intranet, when using the intranet for data transmission, the stability and security of data transmission can be guaranteed, the risk of service interruption can be effectively reduced, and the integrity of the acquisition of operating data can be guaranteed.

[0043] In some optional implementation scenarios, taking the target device as an industrial Internet device as an example, in the network environment provided by the second network, the second server can establish a connection with the target device through the Modbus TCP protocol, thereby achieving the purpose of real-time monitoring of the operating status of the target device. In some optional examples, if the target device includes multiple sub-industrial Internet devices, the second server can establish a connection with each sub-industrial Internet device through the Modbus TCP protocol, so that the operating status of each sub-industrial Internet device can be monitored in real time through the second server, and then the target device can be managed and maintained based on the obtained operating data. The operating status of each sub-industrial Internet device in the target device can be better understood, which is convenient for improving the efficiency of problem location and improving the performance of the target device, so that the target device can be better maintained and managed. For example, the target device is an ore sorting machine, and the multiple sub-industrial Internet devices it includes may include but are not limited to: programmable logic controller (PLC), industrial control computer, blowing device, air conditioner and other sub-industrial Internet devices. In order to determine the operating status of the ore sorting machine, the second server is controlled to be connected to the PLC, industrial control computer, blowing device, air conditioner and other sub-Industrial Internet devices respectively through the Modbus TCP protocol. During the operation of the ore sorting machine, the operating data of each sub-Industrial Internet device can be obtained at the same time. When the first network communication is normal, the first server can obtain the operating update data of each sub-Industrial Internet device through the second server, so as to facilitate the subsequent targeted management and maintenance of each sub-Industrial Internet device.

[0044] Step S220: storing the operation update data through the distributed message service queue system to update the pre-stored historical operation data corresponding to the target device based on the operation update data.

[0045] Distributed message service queue system is a technical architecture for message delivery and communication in distributed systems, with high reliability and scalability, and can effectively store and process large amounts of data, such as text, binary data, or objects.

[0046] In order to improve the fault tolerance of the first server and ensure the storage security of the operation data, the distributed consumption service queue system deployed in the first server is used to store the operation data of the target device, so that other servers or applications can obtain these data in time, ensuring the storage reliability and processing efficiency of the operation data. Among them, the distributed consumption service queue system can include but is not limited to: Kafka Streams (Kafka) system, Rabbit Message Queue (RabbitMQ), Active Message Queue (Apache Active Message Queue, ActiveMQ), which can be selected according to actual needs.

[0047] After obtaining the operation update data, the first server stores the obtained operation update data in the distributed message service queue system, and uses the operation update data to update the pre-stored historical operation data corresponding to the target device, so that the operation data corresponding to the target device stored in the first server can be updated in time and can correspond to the actual operation status of the target device, thereby helping to ensure the accuracy and integrity of the data. Among them, the historical operation data refers to the operation data during the operation of the target device before the operation update data is obtained.

[0048] Step S230: in response to the communication status of the first network returning to normal, determining the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network.

[0049] In response to the communication status of the first network returning to normal, it is characterized that during the communication between the first server and the second server, the first network has experienced an interruption or other abnormal situation, which in turn affects the integrity of the data transmission between the first server and the second server. Therefore, when it is detected that the communication status of the first network has returned to normal, the first server will detect the distributed message service queue system to determine which information to be consumed has not been processed during the period when the first network is disconnected, so that the information to be consumed accumulated during the network disconnection can be discovered and processed in a timely manner to avoid data loss or delayed processing, thereby helping to ensure the performance of the first server. Among them, the information to be consumed is information to be accessed and interacted with the operating data in the distributed message service queue system, and the operating data is the data obtained after the distributed message service queue system was last updated before the communication status of the first network was disconnected.

[0050] Step S240: accessing operation data based on the information to be consumed.

[0051] After determining the information to be consumed, the target operation to be performed on the running data can be determined according to the information to be consumed, and the running data can be accessed and processed according to the target operation, so as to ensure that the previously accumulated information to be consumed can be processed in time after the first network is restored, thereby helping to ensure that the first server can operate normally and improve the accuracy of data management. Among them, the target operation can include but is not limited to: read, write, update, merge, delete, restore and other operations.

[0052] According to the data management method provided by the present invention, a connection is established between the first server and the second server through the first network in advance, and a connection is established between the second server and the target device through the second network, and the stability of the second network is higher than that of the first network, which can ensure that the operation data generated by the target device during operation can be completely obtained by the second server, and then when the communication state of the first network is normal, the first server can obtain the operation update data of the target device from the second server in time, and store it through the distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data, so that the operation data corresponding to the target device stored in the first server can be updated in time, and can correspond to the actual operation of the target device, which helps to ensure the accuracy and integrity of the data. In addition, when it is detected that the communication state of the first network has returned to normal, by determining the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network, it can be clarified which information needs to be processed in time, and then the operation data is accessed based on the information to be consumed, which can ensure the timeliness of the processing of the information to be consumed, and ensure the integrity and reliability of the operation data, which is conducive to improving the performance of the first server and facilitating the subsequent targeted management and maintenance of the target device.

[0053] In some optional embodiments, the above step S210 may include the following steps:

[0054] Step a1: In response to the communication status of the first network being normal, receiving the operation update data of the target device sent by the second server.

[0055] The second server is determined based on the detection result of the data operation log of the target database, the target database is deployed in the second server and corresponds to the target device, and the target database is used to record and save the current operation data of the target device. The data operation log records various operations on the database, including insertion, update and deletion.

[0056] When the communication status of the first network is normal, the second server can detect the data operation log of the target database and analyze the data update status in the target database to obtain the detection result, which provides a basis for the subsequent determination of the operation update data. The detection result can be characterized as the data in the target database is not updated, or it can be characterized as the data in the target database is updated.

[0057] If the detection result indicates that the data in the target database has been updated, the updated data will be used as operation update data and sent to the first server so that the first server can monitor and obtain the target device operation data in real time, discover data updates in time, and achieve the purpose of data synchronization.

[0058] In one example, the data operation log of the target database can be detected by a data capture tool to obtain the detection result. For example, if the target database is a MySQL database, the data change capture tool can be a Debezium plug-in pre-deployed on the MySQL server. The second server can monitor the data operation log (binlog log) of the target database through the Debezium plug-in to detect the data update of the data operation log of the target database, thereby determining and obtaining the operation update data of the target device.

[0059] In some optional embodiments, Figure 3 As shown, the above step S220 may include the following steps S221 to S223:

[0060] Step S221, performing a first encryption process on the running update data to obtain a first encryption process result.

[0061] In order to ensure the security of data transmission, the operation update data is transmitted in an encrypted transmission mode to prevent the operation data from being maliciously stolen or tampered with during the transmission process. Therefore, after the operation update data is determined, the operation update data is first encrypted by a first encryption algorithm to obtain an encrypted first encryption processing result.

[0062] Among them, the first encryption algorithm may include but is not limited to a symmetric encryption algorithm or an asymmetric encryption algorithm. If a symmetric encryption algorithm (for example, AES) is used as the first encryption algorithm, the same key can be used for encryption and decryption during data transmission, which can ensure data transmission security while helping to simplify the encryption / decryption process and improve data processing efficiency. If an asymmetric encryption algorithm (for example, RSA) is used as the first encryption algorithm, it helps to reduce the risk of key leakage and improve data transmission security. Moreover, when an asymmetric encryption algorithm is used for encryption, a digital signature will be generated, which helps to ensure the transmission integrity of the data and the authenticity of the source. The actual use of the first encryption algorithm can be determined according to demand and is not limited here.

[0063] Step S222: The first encryption processing result is stored in the distributed message service queue system, so that the distributed message service queue system updates the pre-stored historical operation data corresponding to the target device based on the first decryption processing result of the first encryption processing result to obtain an update result.

[0064] The distributed message service queue system has the ability to store and process data. Therefore, storing the first encryption processing result in the distributed message service queue system can ensure the storage reliability and update timeliness of the running update data.

[0065] In the distributed message service queue system, historical operation data corresponding to the target device is pre-stored. The historical operation data is operation data that is continuously updated as the target device operates. When the historical operation data needs to be updated, the distributed message service queue system performs a first decryption process on the first encryption process result to obtain the decrypted operation update data, and then merges the operation update data with the corresponding historical operation data to obtain the update data corresponding to the operation status of the target device, thereby realizing the safe update and storage of the target device operation data. The first decryption process corresponding to the first decryption process result corresponds to the first encryption process.

[0066] In some optional examples, the distributed message service queue system may include multiple intermediate storage nodes to improve the fault tolerance of the distributed message service queue system. The process of storing the first encryption processing result through the distributed message service queue system may include: storing the first encryption processing result through multiple intermediate storage nodes respectively, thereby helping to improve the reliability and availability of data storage.

[0067] In other optional examples, the distributed message service queue system can have an intermediate storage node, and when the first encryption processing result is stored through the distributed message service queue system, the first encryption processing result can be stored in the intermediate storage node for centralized management and monitoring, thereby reducing system complexity and maintenance costs.

[0068] Step S223: Cache the update result in the distributed message service queue system through the target search engine.

[0069] The target search engine is a search engine that is connected to a distributed message service queue system and can communicate and interact based on the running data in the distributed message service queue system. The target search engine can be a distributed search engine so as to meet the distributed search and data storage and provide high availability and fault tolerance. For example, the target search engine can be Elasticsearch or other systems that can search for information on the Internet.

[0070] By caching the update results through the target search engine, the demand for access to the operating data of the target device can be improved, and repeated queries to the original data source can be reduced, thereby helping to improve the speed and efficiency of data access.

[0071] In some optional examples, the above step S223 includes:

[0072] Step b1, performing a second encryption process on the update result through the distributed message service queue system to obtain a second encryption result, and sending the second encryption result to the target search engine;

[0073] Step b2: performing a second decryption process on the second encryption result through the target search engine to cache an update result based on the obtained second decryption result.

[0074] Specifically, in order to ensure the security of data transmission, the distributed message service queue system and the target search engine use the second encryption algorithm to encrypt the data during the data transmission process. The second encryption algorithm may be the same as the first encryption algorithm or different from the first encryption algorithm. If the same first encryption algorithm and second encryption algorithm are used, the complexity of implementation can be reduced, which is conducive to improving the efficiency of encryption and decryption. If different first encryption algorithms and second encryption algorithms are used, the security of data transmission can be increased, which is conducive to meeting the use requirements of various scenarios and reducing the risk of data leakage or tampering.

[0075] The update result is subjected to a second encryption process through the distributed message service queue system to obtain a second encryption result, so as to further increase the security of the data, and then the second encryption result is sent to the target search engine. After receiving the second encryption result, the target search engine performs a second decryption process on it, and caches the update result according to the obtained second decryption result for subsequent quick access and use, thereby reducing repeated queries to the original data source and improving the access speed and efficiency of the data. Among them, the second decryption process corresponds to the second encryption result.

[0076] By storing the update data in the above manner, the security and confidentiality of the update results can be guaranteed, while improving the speed and efficiency of data access. In addition, during the transmission process, encryption processing can prevent the update results from being illegally obtained and tampered with, improving the security, reliability and access efficiency of the data, thereby helping to meet a variety of data management needs.

[0077] In some optional embodiments, the above step S230 includes the following steps:

[0078] Step c1, in response to the communication status of the first network returning to normal, the distributed message service queue system is regularly created with a target search engine to determine the unprocessed to-be-consumed information in the distributed message service queue system during the interruption of the first network.

[0079] The distributed message service queue system has a message persistence mechanism. Regardless of whether the network is abnormal, the information to be consumed sent to the distributed message service queue system or the information to be consumed that the distributed message service queue system needs to send out will not be lost.

[0080] In order to ensure the performance of the distributed message service queue system, the snapshot function provided by the target search engine is used to regularly create snapshots of the distributed message service queue system, so that when the communication status of the first network is abnormal, the accumulated information to be consumed during the network interruption can be discovered and processed in time to ensure the normal operation of the system and the accuracy of the data. In one example, the period for creating snapshots of the distributed message service queue system can be created in seconds, which helps to ensure the validity and timeliness of the information to be consumed.

[0081] Since the snapshot is created before the network is interrupted, it can provide information about the system status during the interruption. Therefore, when it is detected that the communication status of the first network has returned to normal, the target search engine can be used to regularly analyze the snapshots created by the distributed message service queue system to determine which messages in the distributed message service queue system have not been processed or consumed during the first network interruption, and then determine the unprocessed information to be consumed during the first network interruption, so as to facilitate timely processing and avoid data loss or delayed processing, thereby helping to improve the reliability and stability of the distributed message service queue system.

[0082] The exemplary embodiments of the present disclosure provide a data management method applied to a first server, such as Figure 4 As shown, the data management method may include the following steps:

[0083] Step S310: In response to the communication status of the first network being normal, obtaining the operation update data of the target device through the second server. Figure 2 Step S210 of the illustrated embodiment will not be described in detail here.

[0084] Step S320: store the operation update data through the distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data. Figure 2 Step S220 of the illustrated embodiment will not be described in detail here.

[0085] Step S330, in response to the communication status of the first network returning to normal, the distributed message service queue system is periodically created using a target search engine to determine the unprocessed to-be-consumed information in the distributed message service queue system during the interruption of the first network.

[0086] Step S340: Access the operation data based on the information to be consumed. Figure 2 Step S240 of the illustrated embodiment will not be described in detail here.

[0087] Step S350: Obtain content update data of the target device from the file management server.

[0088] Among them, the content update data is obtained by the file management server from the second server, and the file management server is connected to the first server and the second server through the first network. The content update data may include but is not limited to media data, files and other data related to the operation of the target device. Since the data format of the content update data is not formatted data, the second server may not be able to directly and effectively obtain it from the second server during the data transmission process with the first server. Therefore, in order to enable the first server to obtain the complete data involved in the operation of the target device, the second server transmits the content data of the target device during the operation process to the file management server after obtaining it. The file management server identifies and extracts the content update data in the content data, and then the first server obtains the content update data of the target device through the file management server to ensure that the first server can obtain the latest content update data of the target device in a timely manner and maintain the synchronization and consistency of the data.

[0089] Step S360: Cache the content of the target search engine to update the data for access by the client of the target search engine.

[0090] The first server sends the acquired content update data to the target search engine. After receiving the content update data, the target search engine caches it for access by the client. The client can obtain the latest content update data by accessing the target search engine, thereby improving the access efficiency and response speed of the data, thereby helping to improve the availability and access speed of the data, and facilitating the client to obtain the latest information in a timely manner.

[0091] According to the data management method provided by the present disclosure, by obtaining content update data of the target device from the file management server and caching it through the target search engine, timely updating and efficient access of data are achieved, thereby providing visitors with a better service experience.

[0092] In some optional embodiments, the operation data can be displayed through a data visualization interface, and the operation data of the target device can be monitored and analyzed in real time according to needs, so that the operation of the target device can be managed and maintained in a targeted manner in a timely manner.

[0093] Based on the same inventive concept, the present disclosure also provides a data management system. Figure 5 As shown, the data management system 100 includes a target device 110 , a second server 120 and a first server 130 .

[0094] The second server 120 is connected to the target device 110 via a second network and is used to record and save the current operation data of the target device 110;

[0095] The first server 130 is connected to the second server 120 through a first network, and the stability of the second network is higher than that of the first network. The first server 130 is used to obtain the operation update data of the target device 110 through the second server 120, and store the operation update data through the distributed message service queue system 131, so as to update the pre-stored historical operation data corresponding to the target device 110 based on the operation update data, determine the unprocessed to-be-consumed information of the distributed message service queue system 131 during the disconnection of the first network, and access the operation data based on the to-be-consumed information, wherein the operation update data is determined based on the current operation data, the to-be-consumed information is the information to be accessed and interacted with the operation data in the distributed message service queue system 131, and the operation data is the data obtained after the distributed message service queue system 131 is last updated before the communication state of the first network is disconnected.

[0096] The specific implementation process of the first server 130 managing the operation data of the target device 110 based on the connection with the second server 120 can be implemented by adopting any one of the above data management method embodiments provided by the present invention, which will not be elaborated here.

[0097] In some optional embodiments, the distributed message service queue system 131 includes multiple intermediate storage nodes, which are deployed on corresponding physical servers respectively, so that when a physical server fails, the intermediate storage nodes on other physical servers can still continue to work, ensuring the reliability and fault tolerance of data transmission, which is beneficial to improving the performance of the distributed message service queue system 131.

[0098] In some optional embodiments, Figure 5 As shown, the first server 130 also includes a target search engine 132, which is used to cache operating data in the distributed message service queue system so that the client 200 of the target search engine 132 can access the operating data of the target device 110 through the target search engine 132 as needed.

[0099] In some optional embodiments, Figure 6 As shown, the data management system 100 also includes: a file management server 140, which is connected to the second server 120 and the first server 130 respectively through the first network. The file management server 140 is used to obtain content update data of the target device 110 from the second server 120, and send the content update data to the target search engine 132, so as to cache the content update data through the target search engine 132 for access by the client 200 of the target search engine 132.

[0100] In some optional implementation scenarios, taking the target device as an ore sorting machine as an example, the process of building a data management system can be as follows:

[0101] The main body of the ore sorting machine and multiple sub-industrial Internet devices are pre-connected to the second server through the second network (intranet), and the Modbus TCP protocol is used to enable the second server to exchange data with the ore sorting machine and the corresponding multiple sub-industrial Internet devices, so that the second server can record and save the operating data of the ore sorting machine.

[0102] The change capture tool (Debezium plug-in) of the distributed message service queue system (kafka) is deployed on the MySQL server, so that when the second server updates the data in the target database in MySQL format, it can detect the data update status of the binlog log based on the data operation log (binlog log) of the target database, and then determine the operation update data of the target device.

[0103] Build a distributed message service queue system (kafka cluster) consisting of three intermediate storage nodes (brokers), and deploy different intermediate storage nodes on different physical servers. Establish a connection between the Debezium plug-in and each intermediate storage node, so that when the target device is running, it can transmit data with the kafka cluster as a producer, so that the kafka cluster can obtain the target device operation update data in time.

[0104] The kafka cluster and the target search engine are integrated on the first server so as to be uniformly managed by the first server. The client of the target search engine may be a consumer of the kafka cluster, and the target search engine may be accessed by a consumer application built by SpringCloud (an ordered set of a series of frameworks). The number of target search engines may be at least one.

[0105] A communication connection is established between the first server and the second server through the first network (external network), and then combined with the ore sorting machine connected to the second server to form the required data management system.

[0106] The process of managing the data of the ore sorting machine through the above data management system can be as follows:

[0107] According to the communication connection between the ore sorter and the second server, the second server records the operation data of the ore sorter and stores it in the internal target database corresponding to the ore sorter. In the second server, the binlog log of the target database is detected for data update through the Debezium plug-in to determine the operation update data of the ore sorter, and when the communication status of the first network is normal, the operation update data is sent to the kafka cluster integrated in the first server.

[0108] The historical operation data of the ore sorting machine during operation is stored in the kafka cluster, and then the historical operation data of the ore sorting machine can be updated in time according to the operation update data of the ore sorting machine transmitted by the Debezium plug-in, thereby ensuring the timeliness of data update. In one example, the Debezium plug-in and the kafka cluster can be encrypted and transmitted using a first encryption algorithm during data transmission.

[0109] The kafka cluster sends the updated operating data of the ore sorter to the target search engine for caching by the target search engine. If the number of target search engines is one, the kafka cluster can directly send the updated operating data to the target search engine. If the number of target search engines is multiple, the kafka cluster can send the updated operating data of the ore sorter to each target search engine separately in the form of broadcast. In one example, during the data transmission process, the kafka cluster and the target search engine can perform encrypted transmission using a second encryption algorithm.

[0110] When the first network communication status is normal, snapshots of the Kafka cluster can be created regularly through the target search engine to clarify the information consumption and processing status of the Kafka cluster.

[0111] In some examples, when the communication status of the first network is normal, the first server can obtain content update data of the target device through the file management server, wherein the content update data is obtained by the file management server from the second server, and the file management server is connected to the first server and the second server through the first network.

[0112] When the first network is interrupted and then returns to normal, the snapshot created by the Kafka cluster can be used to determine the unprocessed information to be consumed by the Kafka cluster during the interruption of the first network, and then targeted processing can be performed through the Kafka cluster to ensure the performance of the data management system.

[0113] In order to enable the client of the target search engine to view the operating data of the ore sorting machine intuitively and quickly, the data can be displayed through a data visualization interface, so that even when the first network is interrupted, the historical operating data of the ore sorting machine can be displayed, thereby helping to timely understand the operating status of the ore sorting machine and facilitate targeted management thereof.

[0114] The data management system and the corresponding data management method provided by the present invention can provide comprehensive support for efficient, secure and intelligent management of modern industrial data transmission, thereby providing a more reliable and efficient solution for industrial data transmission and helping to improve the intelligence level of the industrial production process.

[0115] Based on the same inventive concept, Figure 7 As shown, the present disclosure further provides a data management device 400, which is applied to a first server and includes:

[0116] an acquisition module 410, configured to acquire, in response to the communication state of the first network being normal, operation update data of the target device through the second server, wherein the second server is connected to the target device through the second network, and is configured to record and save the current operation data of the target device, the operation update data is determined based on the current operation data, and the stability of the second network is higher than the stability of the first network;

[0117] The storage module 420 is used to store the operation update data through the distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data;

[0118] A first processing module 430 is used to determine, in response to the communication state of the first network returning to normal, information to be consumed that has not been processed by the distributed message service queue system during the disconnection of the first network, where the information to be consumed is information to be accessed and interacted with the operating data in the distributed message service queue system, and the operating data is data obtained after the distributed message service queue system was last updated before the communication state of the first network was disconnected;

[0119] The second processing module 440 is used to access the operation data based on the information to be consumed.

[0120] In some optional embodiments, the first acquisition module 410 includes: a receiving unit for receiving operation update data of the target device sent by the second server, wherein the second server is determined based on the detection result of the data operation log of the target database, the target database is deployed in the second server and corresponds to the target device, and the target database is used to record and save the current operation data of the target device.

[0121] In some optional embodiments, the storage module 420 includes: a first processing unit, used to perform a first encryption processing on the operation update data to obtain a first encryption processing result; a second processing unit, used to store the first encryption processing result through the distributed message service queue system, so that the distributed message service queue system updates the pre-stored historical operation data corresponding to the target device based on the first decryption processing result of the first encryption processing result to obtain an update result, wherein the first decryption processing corresponding to the first decryption processing result corresponds to the first encryption processing; a third processing unit, used to cache the update result in the distributed message service queue system through the target search engine.

[0122] In some optional embodiments, the distributed message service queue system includes multiple intermediate storage nodes; the second processing unit includes: a storage unit, used to store the first encryption processing results respectively through the multiple intermediate storage nodes.

[0123] In some optional embodiments, the third processing unit includes: a first execution unit, used to perform a second encryption process on the update result through a distributed message service queue system to obtain a second encryption result, and send the second encryption result to a target search engine; a second execution unit, used to perform a second decryption process on the second encryption result through the target search engine to cache the update result according to the obtained second decryption result, wherein the second decryption process corresponds to the second encryption result.

[0124] In some optional embodiments, the first processing module 430 includes: a fourth processing unit, which is used to respond to the communication status of the first network returning to normal, by regularly creating a snapshot of the distributed message service queue system through the target search engine, and determine the unprocessed to-be-consumed information of the distributed message service queue system during the interruption of the first network.

[0125] In some optional embodiments, the apparatus further includes: a second acquisition module, used to acquire content update data of a target device from a file management server, wherein the content update data is acquired by the file management server from a second server, and the file management server is connected to the first server and the second server via a first network; and a third processing module, used to cache the content update data via a target search engine for access by a client of the target search engine.

[0126] In some optional embodiments, the device further includes: a display unit, configured to display the operating data through a data visualization interface.

[0127] Regarding the data management device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0128] Based on the same inventive concept, the present disclosure further provides a computer-readable storage medium, which stores the following program, and the program is used to execute the image segmentation method of any of the aforementioned embodiments.

[0129] The present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or multiple times in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0130] In the context of this application, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0131] Similarly, it should be noted that in order to simplify the description of the disclosure of this application and thus help understand one or more application embodiments, in the above description of the embodiments of this application, multiple features are sometimes merged into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the object of this application are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0132] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is only for example and does not constitute a limitation of the present application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements and amendments to the present application. Such modifications, improvements and amendments are suggested in the present application, so such modifications, improvements and amendments still belong to the spirit and scope of the embodiments of the present application.

Claims

1. A data management method, applied to a first server, comprising: In response to the communication status of the first network being normal, obtaining operation update data of the target device through a second server, wherein the second server is connected to the target device through a second network and is used to record and save current operation data of the target device, the operation update data is determined based on the current operation data, and the stability of the second network is higher than the stability of the first network; and storing the operation update data through a distributed message service queue system to update the pre-stored historical operation data corresponding to the target device based on the operation update data; In response to the communication state of the first network returning to normal, determining information to be consumed that has not been processed by the distributed message service queue system during the disconnection of the first network, the information to be consumed being information to be accessed and interacted with operating data in the distributed message service queue system, the operating data being data obtained after the last update of the distributed message service queue system before the communication state of the first network is disconnected; and, Based on the information to be consumed, the operation data is accessed.

2. The data management method according to claim 1, wherein: The step of acquiring the operation update data of the target device through the second server in response to the communication state of the first network being normal includes: In response to the communication status of the first network being normal, receiving the operation update data of the target device sent by the second server, wherein the second server is determined based on the detection result of the data operation log of the target database, the target database is deployed in the second server and corresponds to the target device, and the target database is used to record and save the current operation data of the target device.

3. The data management method according to claim 2, wherein: The storing the operation update data through the distributed message service queue system to update the pre-stored historical operation data corresponding to the target device based on the operation update data includes: Performing a first encryption process on the operation update data to obtain a first encryption process result; The first encryption processing result is stored by the distributed message service queue system, so that the distributed message service queue system updates the pre-stored historical operation data corresponding to the target device based on the first decryption processing result of the first encryption processing result to obtain an update result, wherein the first decryption processing corresponding to the first decryption processing result corresponds to the first encryption processing; The update result in the distributed message service queue system is cached by the target search engine.

4. The data management method according to claim 3, wherein: The distributed message service queue system includes a plurality of intermediate storage nodes; The storing the first encryption processing result by the distributed message service queue system includes: The first encryption processing result is stored respectively by a plurality of intermediate storage nodes.

5. The data management method according to claim 3 or 4, wherein: The step of caching the update result in the distributed message service queue system through a target search engine includes: Performing a second encryption process on the update result through the distributed message service queue system to obtain a second encryption result, and sending the second encryption result to the target search engine; The second encryption result is subjected to a second decryption process by the target search engine, so as to cache the update result according to the obtained second decryption result, wherein the second decryption process corresponds to the second encryption result.

6. The data management method according to claim 1, wherein: In response to the communication state of the first network returning to normal, determining the unprocessed to-be-consumed information of the distributed message service queue system during the disconnection of the first network includes: In response to the communication status of the first network returning to normal, the target search engine periodically creates snapshots of the distributed message service queue system to determine the unprocessed to-be-consumed information of the distributed message service queue system during the interruption of the first network.

7. The data management method according to claim 6, wherein: The method further comprises: Acquire content update data of the target device from a file management server, wherein the content update data is acquired by the file management server from the second server, and the file management server is connected to the first server and the second server through the first network; The content update data is cached by the target search engine for access by a client of the target search engine.

8. The data management method according to claim 1, wherein: The method further comprises: The operating data is displayed through a data visualization interface.

9. A data management device, applied to a first server, comprising: a first acquisition module, configured to acquire, in response to a normal communication state of the first network, operation update data of the target device through a second server, wherein the second server is connected to the target device through a second network, and is configured to record and save current operation data of the target device, the operation update data is determined based on the current operation data, and the stability of the second network is higher than the stability of the first network; A storage module, configured to store the operation update data through a distributed message service queue system, so as to update the pre-stored historical operation data corresponding to the target device based on the operation update data; A first processing module is configured to determine, in response to the communication state of the first network returning to normal, information to be consumed that has not been processed by the distributed message service queue system during the disconnection of the first network, wherein the information to be consumed is information to be accessed and interacted with operating data in the distributed message service queue system, and the operating data is data obtained after the distributed message service queue system is last updated before the communication state of the first network is disconnected; The second processing module is used to access the operation data based on the information to be consumed.

10. A data management system, comprising: Target device; A second server, connected to the target device via a second network, and configured to record and save current operation data of the target device; A first server is connected to the second server through a first network, the stability of the second network is higher than that of the first network, the first server is used to obtain operation update data of the target device through the second server, store the operation update data through a distributed message service queue system, update the pre-stored historical operation data corresponding to the target device based on the operation update data, determine the unprocessed information to be consumed by the distributed message service queue system during the disconnection of the first network, and access the operation data based on the information to be consumed, wherein the operation update data is determined based on the current operation data, the information to be consumed is information to be accessed and interacted with the operation data in the distributed message service queue system, and the operation data is data obtained after the distributed message service queue system is last updated before the communication state of the first network is disconnected.

11. The data management system according to claim 10, wherein: The distributed message service queue system includes a plurality of intermediate storage nodes, which are respectively deployed on corresponding physical servers.

12. The data management system according to claim 10 or 11, wherein: The first server includes a target search engine, and the target search engine is used to cache the operating data in the distributed message service queue system.

13. The data management system according to claim 12, wherein: The data management system also includes: a file management server, which is connected to the second server and the first server respectively through the first network, and the file management server is used to obtain content update data of the target device from the second server, and send the content update data to the target search engine, so as to cache the content update data through the target search engine for access by the client of the target search engine.

14. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data management method according to any one of claims 1 to 9.

Images