Internal and external network data synchronization method based on Maxwell, RabbitMQ and Internet of Things technology

Through Maxwell software and RabbitMQ message queue technology, the internal and external network data synchronization of the robot inspection platform is realized, solving the complex synchronization problems caused by regional isolation and data security, and achieving low-latency and high-consistency data transmission and synchronization. on a classified robot inspection platform.

CN118802958BActive Publication Date: 2024-11-22北京网藤科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411267587.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-22
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

Due to regional isolation and data security considerations, the existing robot inspection platform cannot communicate directly between the headquarters and the sub-store area, resulting in complex synchronization of internal and external network data and difficult to achieve timeliness.

Method used

Maxwell software is used to monitor the binary log of the child node database, capture the change operation information and convert it into JSON format messages, and transmit it to the Internet of Things cloud platform through the RabbitMQ message queue, and then forward it to the nginx proxy service program by the data copy service program, and finally update the parent node. Database, realizes data synchronization between internal and external networks.

Benefits of technology

Reduces the complexity of data synchronization between internal and external networks of the upper and lower-level platform systems, breaks the barriers between external and internal networks of data synchronization, and allows the parent node database to synchronize data with the child node database in time, with low latency, high consistency, and scalable , reliability and flexibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118802958B_ABST
    Figure CN118802958B_ABST
Patent Text Reader

Abstract

The present application discloses a method for synchronizing intranet and extranet data based on Maxwell, RabbitMQ and Internet of Things technology, and relates to the field of data synchronization technology. The method is that when Maxwell software monitors the binary log of the child node database to be updated, the change operation information of the child node database is captured and converted into a JSON format message, and then the message is forwarded to the child node platform via RabbitMQ in the message production upload and subscription consumption mode, and then the message is forwarded to the data replication service program through the Internet of Things cloud platform, and then the message is forwarded to the nginx proxy service program, the parent node platform and the parent node database in sequence, and finally the parent node database parses the message to obtain the change operation information, and updates the local database according to the change operation information, so that the complexity of intranet and extranet data synchronization of the upper and lower platform systems can be reduced, so that the parent node database in the intranet environment can be synchronized with the child node database in the extranet environment in a timely manner.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data synchronization, and in particular relates to an intranet and extranet data synchronization method based on Maxwell, RabbitMQ and Internet of Things technologies. Background Art

[0002] An intelligent inspection robot is a robot that can perform equipment inspections autonomously or semi-autonomously. It uses sensors and artificial intelligence algorithms to monitor, diagnose and maintain equipment. Therefore, it can not only replace manual repetitive, long-term and dangerous / remote work, improve inspection efficiency and reduce safety risks, but also monitor equipment status in real time, detect abnormal situations in a timely manner, and through data analysis and learning, more accurately judge equipment failures and performance degradations, and avoid missed detections and misjudgments.

[0003] Due to system business reasons caused by special conditions, the existing robot inspection platform has the following data synchronization problems:

[0004] (1) The robot inspection platform is divided into two levels: headquarters and sub-warehouse area. The servers of the headquarters and sub-warehouse area are not in the same place. The headquarters server may be located in Beijing, and the sub-warehouse area server may be located in Jinzhou, which results in geographical isolation. At the same time, for data security reasons, the platform's data storage will not be cloud-based, so the headquarters and sub-warehouse area servers do not use the same database, but the functions of the headquarters platform rely on the data of the sub-warehouse area platform;

[0005] (2) Since the customer has strict requirements on the deployment environment of the headquarters platform, it is divided into an intranet environment and an extranet environment. The services in the intranet environment can only communicate with another service in the same LAN. In this case, it cannot directly communicate with the extranet environment (such as the Alibaba Cloud IoT platform). Considering that such a design cannot communicate with the extranet environment, the customer can only allow services deployed on the DMZ (abbreviation of Demilitarized Zone, the Chinese name is isolation zone, also known as demilitarized zone, which is a buffer between non-secure systems and secure systems set up to solve the problem that external network access users cannot access internal network servers after installing a firewall) servers to communicate with the extranet environment, while services deployed on the DMZ server need to communicate with the intranet environment through the nginx proxy. At this time, the intermediate link of data synchronization becomes more complicated. Summary of the invention

[0006] The purpose of the present invention is to provide an internal and external network data synchronization method based on Maxwell, RabbitMQ and Internet of Things technology, so as to solve the problems of complex internal and external network data synchronization and difficulty in timely synchronization of the upper and lower databases due to barriers in existing upper and lower level platform systems such as robot inspection platforms.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In the first aspect, a method for synchronizing intranet and extranet data based on Maxwell, RabbitMQ and Internet of Things technology is provided, including:

[0009] Start the Maxwell software to monitor the binary log of the subnode database, wherein the Maxwell software and the subnode database are both deployed on the server of the subnode;

[0010] When the Maxwell software monitors that the binary log of the subnode database is updated, it captures the change operation information of the subnode database and converts the change operation information into a JSON format message;

[0011] The Maxwell software sends the JSON format message to the RabbitMQ software and to a message queue pre-bound to the Maxwell software used as a message producer, wherein the RabbitMQ software is also deployed on the server of the child node;

[0012] The subnode platform subscribes to and obtains the JSON format message from the RabbitMQ software, wherein the subnode platform is also deployed on the server of the subnode and is bound to the message queue as a message consumer;

[0013] The subnode platform transmits the JSON format message to the IoT cloud platform in the name of the IoT server;

[0014] The data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform in the name of the Internet of Things client, wherein the data replication service program is deployed on an isolation zone server, the isolation zone server is located in the external network environment of the parent node, and the parent node and the child node are a pair of nodes with a superior-subordinate relationship;

[0015] The data replication service program sends the JSON format message to the nginx proxy service program, wherein the nginx proxy service program is deployed on the server of the parent node, and the server of the parent node is located in the intranet environment of the parent node;

[0016] The nginx proxy service program sends the JSON format message to the parent node platform, wherein the parent node platform is also deployed on the server of the parent node;

[0017] The parent node platform sends the JSON format message to a parent node database, wherein the parent node database is also deployed on a server of the parent node;

[0018] The parent node database parses the JSON format message to obtain the change operation information, and updates the local database according to the change operation information.

[0019] Based on the above invention content, a new solution for synchronizing intranet and extranet data based on Maxwell, RabbitMQ and Internet of Things technology is provided, that is, when Maxwell software monitors that the binary log of the child node database is updated, it captures the change operation information of the child node database and converts it into a JSON format message, and then forwards the message to the child node platform via RabbitMQ in the message production upload and subscription consumption mode, and then forwards the message to the data replication service program through the Internet of Things cloud platform, and then forwards the message to the nginx proxy service program, the parent node platform and the parent node database in sequence, and finally the parent node database parses the message to obtain the change operation information, and updates the local database according to the change operation information, so that the complexity of intranet and extranet data synchronization of the upper and lower platform systems can be reduced, the barrier between the extranet and intranet data synchronization can be broken, so that the parent node database in the intranet environment can be synchronized with the child node database in the extranet environment in a timely manner, which is particularly suitable for robot inspection platforms with two levels of headquarters and sub-warehouse areas, and is convenient for practical application and promotion.

[0020] In a possible design, the parent node and the child node correspond one-to-one to two levels of the headquarters and the sub-warehouse area in the robot inspection platform.

[0021] In a possible design, the sub-node platform is used to:

[0022] Configure and manage robot information of inspection robots used to perform inspection tasks in the warehouse area;

[0023] And / or, configure and manage the inspection routes, inspection strategies, inspection tasks, inspection records and / or inspection points required by the current patrol robot when performing the inspection tasks in the corresponding warehouse area;

[0024] And / or, configure and manage the system for judging whether the inspection data generated by the inspection robot in the process of performing the inspection task in the warehouse area to which it belongs meets the alarm standard, and generate an alarm record when the alarm standard is met and / or support the processing strategy of the inspection robot when receiving the alarm data, wherein the alarm record supports manual review to prevent the robot from misjudging, and the processing strategy includes one-key emergency stop, resumption of inspection and / or immediate return;

[0025] And / or, support viewing the robot cockpit and / or the depot cockpit, wherein the robot cockpit is used to display the robot overview, alarm information, process indicator results, real-time video obtained by collecting visible light by the robot camera, real-time video obtained by collecting infrared light by the robot camera and / or artificial intelligence analysis results of video data, and the depot cockpit is used to display the robot list of the depot area, the number of inspections on the day / week / month, the details of the inspection tasks on the day, the number of alarms on the day / week / month, TopN alarm events and / or oil depot alarm conditions;

[0026] And / or, it supports viewing the task statistics, alarm statistics, alarm trends, alarm levels, alarm type analysis results and / or TopN alarm inspection points of the robots in the warehouse area with a time span of weeks / months / years.

[0027] In a possible design, the parent node platform is used to:

[0028] Supports viewing the headquarters cockpit, where the headquarters cockpit is used to display the number of robots in all sub-areas grouped by status, inspection statistics, real-time inspection results, alarm information, alarm trends and / or real-time alarm events;

[0029] And / or, support query and filter of robot inspection records of all sub-warehouse areas, and export inspection reports;

[0030] and / or, support query and filter of robot alarm records in all sub-areas;

[0031] And / or, support report analysis based on the business data of all sub-database areas, wherein the report analysis includes task statistics, alarm statistics, alarm trend estimation, alarm level classification, alarm analysis, alarm distribution analysis, alarm analysis of each data base area and / or TopN alarm type sorting.

[0032] In one possible design, the child node database or the parent node database is a MySQL database.

[0033] In a possible design, the change operation information includes a change operation type, a change operation table name and a change operation data content, wherein the change operation type includes an insert operation, an update operation and / or a delete operation, and the change operation data content includes new data for an insert operation, old data before the update and new data after the update for an update operation and / or deleted data for a delete operation.

[0034] In one possible design, the IoT cloud platform is the Alibaba Cloud IoT platform, and the access document of the Alibaba Cloud IoT platform is referenced in advance, and the sub-node platform is registered as a server to the Alibaba Cloud IoT platform so that the sub-node platform is connected to the Alibaba Cloud IoT platform and has a topic with subscription permissions, and the data replication service program is also registered as a client to the Alibaba Cloud IoT platform so that the data replication service program is connected to the Alibaba Cloud IoT platform and has a topic with subscription permissions.

[0035] In a possible design, the subnode platform transmits the JSON format message to the IoT cloud platform in the name of the IoT server, including:

[0036] The subnode platform transmits the JSON format message to the IoT cloud platform in the name of the IoT server through the HTTPS protocol;

[0037] The data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform on behalf of the Internet of Things client, including:

[0038] The data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform in the name of the Internet of Things client through the MQTT protocol, wherein the data replication service program is deployed on the isolation area server, and the isolation area server is located in the external network environment of the parent node, and the parent node and the child node are a pair of nodes with a superior-subordinate relationship.

[0039] In a possible design, the data replication service program sends the JSON format message to the nginx proxy service program, including:

[0040] The data replication service program sends the JSON format message to the nginx proxy service program through the HTTPS protocol, wherein the nginx proxy service program is deployed on the server of the parent node, and the server of the parent node is located in the intranet environment of the parent node.

[0041] In a possible design, updating the local database according to the change operation information includes:

[0042] The change operation information is encapsulated into a class corresponding to the target table, and then the local database is updated with the help of the MybatisPlus framework.

[0043] Beneficial effects of the above scheme:

[0044] (1) The present invention creatively provides a new solution for synchronizing intranet and extranet data based on Maxwell, RabbitMQ and IoT technology, that is, when Maxwell software monitors the binary log of the child node database for update, it captures the change operation information of the child node database and converts it into a JSON format message, and then forwards the message to the child node platform via RabbitMQ in the form of message production upload and subscription consumption, and then forwards the message to the data replication service program through the IoT cloud platform, and then forwards the message to the nginx proxy service program, the parent node platform and the parent node database in sequence, and finally the parent node database parses the message to obtain the change operation information, and updates the local database according to the change operation information, so as to reduce the complexity of intranet and extranet data synchronization of the upper and lower platform systems, break the barrier of extranet and intranet data synchronization, and enable the parent node database in the intranet environment to synchronize data with the child node database in the extranet environment in a timely manner, which is particularly suitable for robot inspection platforms with two levels of headquarters and sub-warehouse areas, and is convenient for practical application and promotion;

[0045] (2) It also has the feature of low latency, that is, Maxwell can capture changes in the MySQL database in real time and quickly send these changes to RabbitMQ, achieving low-latency data transmission, making Maxwell provide higher real-time performance compared to traditional batch data synchronization tools;

[0046] (3) It also has the characteristic of high data consistency. That is, by reading MySQL binlog, Maxwell can ensure that the captured change data is highly consistent with the actual data in the database, while traditional data synchronization methods may cause data inconsistency due to delays or other reasons;

[0047] (4) It also has the characteristic of horizontal scalability. That is, RabbitMQ, as a message queue system, supports horizontal expansion and can handle a large number of concurrent messages, thereby improving the system throughput.

[0048] (5) It also has the feature of system decoupling, that is, through the message queue, the producer (Maxwell) and the consumer (the application that processes JSON messages) are decoupled, which facilitates the maintenance and expansion of the system;

[0049] (6) It also has the feature of message persistence, that is, RabbitMQ supports message persistence, ensuring that messages will not be lost in the event of a system failure, thereby improving system reliability;

[0050] (7) It also supports multiple consumers, that is, multiple consumers can read messages from RabbitMQ and perform different processing operations, such as data analysis and monitoring alarms, to improve the flexibility of the system, while traditional methods may have limitations in scalability and flexibility;

[0051] (8) It also has the feature of simplified architecture. That is, Maxwell directly reads the binary log binlog of the MySQL database, without the need for additional development and maintenance at the application level, which simplifies the system architecture. Traditional methods may require adding additional logic at the application level to capture and synchronize data changes.

[0052] (9) It is also open source and community-supported. That is, Maxwell is an open source project with active community support, and users can customize and expand it according to their own needs. In contrast, some commercial data synchronization tools may have limitations in customization and expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of an intranet and extranet data synchronization method based on Maxwell, RabbitMQ and Internet of Things technologies provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structures of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these embodiments without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0056] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of the exemplary embodiments of the present invention.

[0057] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can indicate three situations: A exists alone, B exists alone, or A and B exist at the same time. For another example, A, B and / or C can indicate the existence of any one of A, B and C or any combination of them. The term " / and" that may appear in this document describes another type of association object relationship, indicating that there may be two relationships. For example, A / and B can indicate two situations: A exists alone or A and B exist at the same time. In addition, the character " / " that may appear in this document generally indicates that the previous and next associated objects are in an "or" relationship.

[0058] Example

[0059] like Figure 1 As shown, the internal and external network data synchronization method provided in this embodiment and based on Maxwell, RabbitMQ and Internet of Things technology includes but is not limited to the following steps S1 to S10.

[0060] S1. Start the Maxwell software to monitor the binary log of the subnode database, wherein the Maxwell software and the subnode database are both deployed on the server of the subnode.

[0061] In step S1, the Maxwell software is a real-time crawling software for MySQL (an open source small relational database management system) database, which is open sourced by Zendesk in the United States and written in Java. It can read the binary log Binlog in the MySQL database in real time, and generate messages in JSON (JavaScript Object Notation, JavaScript object representation, which is designed based on a subset of ECMAScript and is an open standard file format and data exchange format, which is easy for people to read and write, and easy for machines to parse and generate) format, and send it as a producer to applications such as Kafka (an open source stream processing platform developed by the Apache Software Foundation, written in Scala and Java), Kinesis (a platform for streaming data on the Amazon cloud, which allows customers to easily load and analyze streaming data, and also allows customers to build custom streaming data applications according to specific needs), RabbitMQ (an open source message agent software that implements the Advanced Message Queuing Protocol, written in Erlang language), files or other platforms. The sub-node database is specifically a MySQL database. In order to use the Maxwell software to monitor the binary log of the subnode database, the following environment preparation and software configuration are required: (1) Install MySQL: Ensure that the MySQL database has been installed and the binary log (binlog) has been configured; (2) Install Maxwell: Download and configure Maxwell (i.e., configure the necessary parameters for Maxwell to connect to the MySQL database, such as the host address, port, user name and password, etc.), and ensure that it can connect to the MySQL database and read the binlog. In addition, the subnode can be, but is not limited to, the sub-store area corresponding to the robot inspection platform.

[0062] S2. When the Maxwell software monitors that the binary log of the subnode database is updated, it captures the change operation information of the subnode database and converts the change operation information into a JSON format message.

[0063] In the step S2, when the user modifies the data in the MySQL database through human-computer interaction, the MySQL database will write these data changes into the binary log, so the Maxwell software can timely monitor the data in the subnode database through the binary log update event, and then timely capture the change operation information of the subnode database and convert it into JSON format. Specifically, the change operation information includes but is not limited to the change operation type, the change operation table name and the change operation data content, wherein the change operation type includes but is not limited to insert operation, update operation and / or delete operation, and the change operation data content includes but is not limited to the new data of the insert operation, the old data before the update of the update operation and the new data after the update and / or the deleted data of the delete operation.

[0064] S3. The Maxwell software sends the JSON format message to the RabbitMQ software and to the message queue pre-bound to the Maxwell software used as a message producer, wherein the RabbitMQ software is also deployed on the server of the child node.

[0065] In step S3, the RabbitMQ software is an open source message agent software that implements the Advanced Message Queuing Protocol. In order to achieve the purpose of message uploading, the following environment preparation and software configuration are required: (1) Install RabbitMQ: Install and configure the RabbitMQ message queue system; (2) Set the output target of Maxwell to RabbitMQ, including the host address, port, user name, password and queue name of RabbitMQ.

[0066] S4. The subnode platform subscribes to and obtains the JSON format message from the RabbitMQ software, wherein the subnode platform is also deployed on the server of the subnode and is bound to the message queue as a message consumer.

[0067] In step S4, the subnode platform is used to establish a connection and channel with RabbitMQ as a RabbitMQ consumer role, and a message processing callback function is predefined to subscribe to consumption messages. When the subnode specifically corresponds to the sub-warehouse area level in the robot inspection platform, the subnode platform is specifically but not limited to: configuring and managing the robot information of the inspection robot used to perform the inspection task of the warehouse area; and / or, configuring and managing the inspection routes, inspection strategies, inspection tasks, inspection records and / or inspection points required by the current patrol robot when performing the inspection task of the warehouse area; and / or, configuring and managing the inspection data generated by the inspection robot in the process of performing the inspection task of the warehouse area to which it belongs meets the alarm standard, and generating an alarm record when the alarm standard is met and / or supporting the processing strategy of the inspection robot when receiving the alarm data, wherein the alarm record supports manual review to prevent the robot from misjudging, and the processing strategy includes but is not limited to one-key emergency stop, resumption of inspection and / or immediate return, etc.; and / or, supporting viewing the robot cockpit and / or warehouse area driving Cabin, wherein the robot cockpit is used for but not limited to displaying the robot overview, alarm information, process indicator results, real-time video obtained by collecting visible light by the robot camera, real-time video obtained by collecting infrared light by the robot camera and / or artificial intelligence analysis results of video data, etc., and the depot cockpit is used for but not limited to displaying the robot list of the depot area, the number of inspections on the same day / week / month, the inspection task details of the day, the number of alarms on the same day / week / month, TopN alarm events (for example, the top five alarm events with the highest frequency, i.e., Top5 alarm events, N is a positive integer and the example value is 5) and / or oil depot alarm conditions, etc.; and / or, supporting viewing the task statistics, alarm statistics, alarm trends, alarm levels, alarm type analysis results and / or TopN alarm inspection points (for example, the top five alarm inspection points with the highest frequency) of the robots in the depot area with a time span of weeks / months / years.

[0068] S5. The sub-node platform transmits the JSON format message to the Internet of Things cloud platform in the name of the Internet of Things server.

[0069] In step S5, the IoT cloud platform is specifically but not limited to the Alibaba Cloud IoT platform, and the access document of the Alibaba Cloud IoT platform is referred to in advance, and the sub-node platform is registered as a server to the Alibaba Cloud IoT platform, so that the sub-node platform is connected to the Alibaba Cloud IoT platform and has a topic with subscription rights. In order to achieve the purpose of uploading messages to the IoT cloud platform, the following environment preparation and software configuration are required: (1) Alibaba Cloud IoT platform configuration: first open an enterprise instance, then create a product, that is, log in to the Alibaba Cloud IoT platform console, create a new product, and fill in information such as the product name and node type, record the product key ProductKey and password ProductSecret, define the product model and the topic with subscription rights; finally create a device, that is, create a new device under the newly created product, and record the device name DeviceName and password DeviceSecret; (2) Generate a signature: Alibaba Cloud IoT platform requires the use of HMAC-SHA256 signature to verify the legitimacy of the request. The signature generation steps are as follows: construct a signature string and use the device key to perform HMAC-SHA256 signature. Specifically, the sub-node platform transmits the JSON format message to the IoT cloud platform in the name of the IoT server through the HTTPS (HyperText Transfer Protocol Secure, a transmission protocol for secure communication over a computer network), that is, the sub-node platform can send an HTTPS request to send the JSON format message to the topic with subscription rights. In addition, the sub-node platform preferably processes the response in the following manner: (1) parsing the response, that is, processing the response returned by the Alibaba Cloud IoT platform and determining whether the message is successfully transmitted based on the response content; (2) error handling, that is, adding an error handling mechanism when sending a message to ensure that it can be properly handled and recovered when an error occurs.

[0070] S6. The data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform in the name of the Internet of Things client, wherein the data replication service program is deployed on the isolation area server, the isolation area server is located in the external network environment of the parent node, and the parent node and the child node are a pair of nodes with a superior-subordinate relationship.

[0071] In step S6, the above-mentioned data replication service (DRS) is an easy-to-use, stable, efficient cloud service for real-time database migration and real-time database synchronization; in this embodiment, the data replication service program is used as an intermediate service for data synchronization to solve the problem that the headquarters platform cannot directly communicate with the external network environment. In order to achieve the purpose of message subscription acquisition, the following environment preparation and software configuration are required: (1) The data replication service program is connected to the Alibaba Cloud Internet of Things platform through the MQTT (Message Queuing Telemetry Transport) protocol, wherein the connection information specifically includes: the client unique identifier, the address and port of the enterprise instance, the maximum time interval between sending / receiving messages (in seconds), the client password, the timeout time for the client network connection success / failure (in seconds), the product name and the device name, etc. Specifically, the data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform in the name of the Internet of Things client and through the MQTT protocol. In addition, when the IoT cloud platform is specifically the Alibaba Cloud IoT platform, it is also necessary to refer to the access document of the Alibaba Cloud IoT platform in advance and register the data replication service program as a client to the Alibaba Cloud IoT platform so that the data replication service program is connected to the Alibaba Cloud IoT platform and has subscription permissions for the topic.

[0072] In the step S6, specifically, the parent node may be but is not limited to corresponding to the headquarters level in the robot inspection platform, that is, the parent node and the child node correspond one-to-one to the headquarters and sub-warehouse level in the robot inspection platform.

[0073] S7. The data replication service program sends the JSON format message to the nginx proxy service program, wherein the nginx proxy service program is deployed on the server of the parent node, and the server of the parent node is located in the intranet environment of the parent node.

[0074] In step S7, the above-mentioned nginx proxy service is a high-performance HTTP (Hypertext Transfer Protocol) and reverse proxy web (World Wide Web, also known as the World Wide Web) service. In order to achieve the purpose of message forwarding, the following environmental preparation and software configuration are required: (1) on the server of the parent node, download and install nginx; (2) configure nginx to support HTTPS, that is, first create a self-signed SSL (Secure socket layer) certificate, then edit the nginx configuration file, set HTTPS support, and finally test and restart the nginx service. Specifically, the data replication service program sends the JSON format message to the nginx proxy service program through the HTTPS protocol.

[0075] S8. The nginx proxy service program sends the JSON format message to the parent node platform, wherein the parent node platform is also deployed on the server of the parent node.

[0076] In the step S8, when the parent node specifically corresponds to the headquarters level in the robot inspection platform, the parent node platform is specifically but not limited to: supporting viewing of the headquarters cockpit, wherein the headquarters cockpit is used for but not limited to displaying the number of robots in all sub-warehouse areas grouped according to status, inspection statistics, real-time inspection results, alarm information, alarm trends and / or real-time alarm events, etc.; and / or, supporting query and filtering of robot inspection records of all sub-warehouse areas, and exporting inspection reports; and / or, supporting query and filtering of robot alarm records of all sub-warehouse areas; and / or, supporting report analysis based on business data of all sub-warehouse areas, wherein the report analysis includes but is not limited to task statistics, alarm statistics, alarm trend estimation, alarm level division, alarm analysis, alarm distribution analysis, alarm analysis of each warehouse area and / or TopN alarm type sorting (for example, sorting of the top five alarm types with the highest frequency), etc.

[0077] S9. The parent node platform sends the JSON format message to the parent node database, wherein the parent node database is also deployed on the server of the parent node.

[0078] In step S9, the parent node database is specifically but not limited to a MySQL database. In order to achieve the purpose of receiving the JSON format message, the following environment preparation and software configuration are required: (1) Install and configure the MySQL database in the headquarters platform intranet environment; (2) Create the corresponding database and table.

[0079] S10. The parent node database parses the JSON format message to obtain the change operation information, and updates the local database according to the change operation information.

[0080] In the step S10, specifically, updating the local database according to the change operation information includes but is not limited to: encapsulating the change operation information into a class corresponding to the target table (which is determined by the change operation table name in the change operation information), and then updating the local database with the help of the MybatisPlus framework. The MybatisPlus framework is an enhancement tool of MyBatis (which is an excellent and java-based persistence layer framework, which encapsulates jdbc internally, so that developers only need to focus on the sql statement itself, and do not need to spend energy to deal with the complicated processes such as loading drivers, creating connections and creating statements). It is based on MyBatis and only enhances it without making changes, which is born to simplify development and improve efficiency; the MybatisPlus framework provides many general CRUD (that is, the abbreviation of the initials of several words such as Create, Read, Update and Delete when doing calculation processing) methods, such as insert, update, delete and query, so the synchronous update of the parent node database can be completed according to the change operation information.

[0081] In summary, the intranet and extranet data synchronization method provided by this embodiment has the following technical effects:

[0082] (1) This embodiment provides a new solution for synchronizing intranet and extranet data based on Maxwell, RabbitMQ and IoT technology. That is, when Maxwell software monitors the binary log of the child node database for update, it captures the change operation information of the child node database and converts it into a JSON format message. Then, the message is forwarded to the child node platform via RabbitMQ in the form of message production, upload and subscription consumption. Then, the message is forwarded to the data replication service program through the IoT cloud platform. Then, the message is forwarded to the nginx proxy service program, the parent node platform and the parent node database in sequence. Finally, the parent node database parses the message to obtain the change operation information and updates the local database according to the change operation information. In this way, the complexity of intranet and extranet data synchronization of the upper and lower platform systems can be reduced, and the barriers to data synchronization between the extranet and the intranet can be broken, so that the parent node database in the intranet environment can synchronize data with the child node database in the extranet environment in a timely manner. This is particularly suitable for robot inspection platforms with two levels, namely, headquarters and sub-warehouse areas, and is convenient for practical application and promotion.

[0083] (2) It also has the feature of low latency, that is, Maxwell can capture changes in the MySQL database in real time and quickly send these changes to RabbitMQ, achieving low-latency data transmission, making Maxwell provide higher real-time performance compared to traditional batch data synchronization tools;

[0084] (3) It also has the characteristic of high data consistency. That is, by reading MySQL binlog, Maxwell can ensure that the captured change data is highly consistent with the actual data in the database, while traditional data synchronization methods may cause data inconsistency due to delays or other reasons;

[0085] (4) It also has the characteristic of horizontal scalability. That is, RabbitMQ, as a message queue system, supports horizontal expansion and can handle a large number of concurrent messages, thereby improving the system throughput.

[0086] (5) It also has the feature of system decoupling, that is, through the message queue, the producer (Maxwell) and the consumer (the application that processes JSON messages) are decoupled, which facilitates the maintenance and expansion of the system;

[0087] (6) It also has the feature of message persistence, that is, RabbitMQ supports message persistence, ensuring that messages will not be lost in the event of a system failure, thereby improving system reliability;

[0088] (7) It also supports multiple consumers, that is, multiple consumers can read messages from RabbitMQ and perform different processing operations, such as data analysis and monitoring alarms, to improve the flexibility of the system, while traditional methods may have limitations in scalability and flexibility;

[0089] (8) It also has the feature of simplified architecture. That is, Maxwell directly reads the binary log binlog of the MySQL database, without the need for additional development and maintenance at the application level, which simplifies the system architecture. Traditional methods may require adding additional logic at the application level to capture and synchronize data changes.

[0090] (9) It is also open source and community-supported. That is, Maxwell is an open source project with active community support, and users can customize and expand it according to their own needs. In contrast, some commercial data synchronization tools may have limitations in customization and expansion.

[0091] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for synchronizing intranet and extranet data based on Maxwell, RabbitMQ and Internet of Things technology, characterized in that: include: Start the Maxwell software to monitor the binary log of the subnode database, wherein the Maxwell software and the subnode database are both deployed on the server of the subnode; When the Maxwell software monitors that the binary log of the subnode database is updated, it captures the change operation information of the subnode database and converts the change operation information into a JSON format message; The Maxwell software sends the JSON format message to the RabbitMQ software and to a message queue pre-bound to the Maxwell software used as a message producer, wherein the RabbitMQ software is also deployed on the server of the child node; The subnode platform subscribes to and obtains the JSON format message from the RabbitMQ software, wherein the subnode platform is also deployed on the server of the subnode and is bound to the message queue as a message consumer; The subnode platform transmits the JSON format message to the IoT cloud platform in the name of the IoT server; The data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform in the name of the Internet of Things client, wherein the data replication service program is deployed on an isolation zone server, the isolation zone server is located in the external network environment of the parent node, and the parent node and the child node are a pair of nodes with a superior-subordinate relationship; The data replication service program sends the JSON format message to the nginx proxy service program, wherein the nginx proxy service program is deployed on the server of the parent node, and the server of the parent node is located in the intranet environment of the parent node; The nginx proxy service program sends the JSON format message to the parent node platform, wherein the parent node platform is also deployed on the server of the parent node; The parent node platform sends the JSON format message to a parent node database, wherein the parent node database is also deployed on a server of the parent node; The parent node database parses the JSON format message to obtain the change operation information, and updates the local database according to the change operation information.

2. The method for synchronizing data between an intranet and an extranet according to claim 1, wherein: The parent node and the child node correspond one-to-one to the headquarters and sub-warehouse area levels in the robot inspection platform.

3. The method for synchronizing data between an internal and external network as claimed in claim 2, characterized in that: The sub-node platform is used for: Configure and manage robot information of inspection robots used to perform inspection tasks in the warehouse area; And / or, configure and manage the inspection routes, inspection strategies, inspection tasks, inspection records and / or inspection points required by the current patrol robot when performing the inspection tasks in the corresponding warehouse area; And / or, configure and manage the system for judging whether the inspection data generated by the inspection robot in the process of performing the inspection task in the warehouse area to which it belongs meets the alarm standard, and generate an alarm record when the alarm standard is met and / or support the processing strategy of the inspection robot when receiving the alarm data, wherein the alarm record supports manual review to prevent the robot from misjudging, and the processing strategy includes one-key emergency stop, resumption of inspection and / or immediate return; And / or, support viewing the robot cockpit and / or the depot cockpit, wherein the robot cockpit is used to display the robot overview, alarm information, process indicator results, real-time video obtained by collecting visible light by the robot camera, real-time video obtained by collecting infrared light by the robot camera and / or artificial intelligence analysis results of video data, and the depot cockpit is used to display the robot list of the depot area, the number of inspections on the day / week / month, the details of the inspection tasks on the day, the number of alarms on the day / week / month, TopN alarm events and / or oil depot alarm conditions; And / or, it supports viewing the task statistics, alarm statistics, alarm trends, alarm levels, alarm type analysis results and / or TopN alarm inspection points of the robots in the warehouse area with a time span of weeks / months / years.

4. The method for synchronizing data between an internal and external network as claimed in claim 2, characterized in that: The parent node platform is used for: Supports viewing the headquarters cockpit, where the headquarters cockpit is used to display the number of robots in all sub-areas grouped by status, inspection statistics, real-time inspection results, alarm information, alarm trends and / or real-time alarm events; And / or, support query and filter of robot inspection records of all sub-warehouse areas, and export inspection reports; and / or, support query and filter of robot alarm records in all sub-areas; And / or, support report analysis based on the business data of all sub-database areas, wherein the report analysis includes task statistics, alarm statistics, alarm trend estimation, alarm level classification, alarm analysis, alarm distribution analysis, alarm analysis of each data base area and / or TopN alarm type sorting.

5. The method for synchronizing data between an internal and external network as claimed in claim 1, characterized in that: The child node database or the parent node database is a MySQL database.

6. The method for synchronizing data between an internal and external network as claimed in claim 1, characterized in that: The change operation information includes the change operation type, the change operation table name and the change operation data content, wherein the change operation type includes an insert operation, an update operation and / or a delete operation, and the change operation data content includes new data for an insert operation, old data before the update and new data after the update for an update operation and / or deleted data for a delete operation.

7. The method for synchronizing data between an internal and external network as claimed in claim 1, characterized in that: The IoT cloud platform is the Alibaba Cloud IoT platform, and the access document of the Alibaba Cloud IoT platform is referenced in advance, and the sub-node platform is registered as a server to the Alibaba Cloud IoT platform so that the sub-node platform is connected to the Alibaba Cloud IoT platform and has a topic with subscription permissions, and the data replication service program is also registered as a client to the Alibaba Cloud IoT platform so that the data replication service program is connected to the Alibaba Cloud IoT platform and has a topic with subscription permissions.

8. The method for synchronizing data between an internal and external network as claimed in claim 1, characterized in that: The subnode platform transmits the JSON format message to the IoT cloud platform in the name of the IoT server, including: The subnode platform transmits the JSON format message to the IoT cloud platform in the name of the IoT server through the HTTPS protocol; The data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform on behalf of the Internet of Things client, including: The data replication service program receives the JSON format message forwarded by the Internet of Things cloud platform in the name of the Internet of Things client through the MQTT protocol, wherein the data replication service program is deployed on the isolation area server, and the isolation area server is located in the external network environment of the parent node, and the parent node and the child node are a pair of nodes with a superior-subordinate relationship.

9. The method for synchronizing data between an internal and external network as claimed in claim 1, characterized in that: The data replication service program sends the JSON format message to the nginx proxy service program, including: The data replication service program sends the JSON format message to the nginx proxy service program through the HTTPS protocol, wherein the nginx proxy service program is deployed on the server of the parent node, and the server of the parent node is located in the intranet environment of the parent node.

10. The method for synchronizing data between an internal and external network as claimed in claim 1, characterized in that: Updating the local database according to the change operation information includes: The change operation information is encapsulated into a class corresponding to the target table, and then the local database is updated with the help of the MybatisPlus framework.