A distributed spatiotemporal database data synchronization system and method based on message queue

Abstract

The present invention discloses a distributed spatiotemporal database data synchronization system and method based on a message queue, and relates to the field of distributed management technology of spatiotemporal data, wherein the system includes a producer, a message queue, a consumer group, and a multi-node distributed database; the producer identifies the spatial range of the spatiotemporal data message, and delivers the spatiotemporal data message to the spatial partition of the specified message queue; multiple consumer groups pull the messages of the spatial partition in the message queue, perform time classification on the spatiotemporal data messages, and store the spatiotemporal data messages in the multi-node distributed database with spatiotemporal dimension features. Based on the multi-topic subscription and broadcast consumption capabilities of the message queue, the present invention proposes a method for distributed storage of spatiotemporal data in the spatiotemporal dimension and multi-node synchronous backup, produces messages based on the spatiotemporal dimension and delivers them to a specific message queue topic, pushes data to multiple nodes based on broadcast consumption, and realizes the distributed storage and synchronous backup of spatiotemporal data in the spatiotemporal dimension.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed spatiotemporal data management, and in particular to a distributed spatiotemporal database data synchronization system and method based on a message queue. Background Art

[0002] As the satellite launch frequency at home and abroad increases year by year, the number of imaging satellites is increasing, and the satellite image data acquired increases by TB level every day. A large amount of spatiotemporal data is continuously stored in the database, which brings challenges to the multi-dimensional distributed storage of data and multi-node data synchronization.

[0003] Spatiotemporal data comes from a wide range of sources, often involving multiple manufacturers and organizations. The data information varies significantly and the data volume is huge. While managing these massive amounts of data, it is also necessary to ensure the stability and efficiency of the system, ensure the security and reliability of the data, and be able to achieve efficient and fast data queries to ensure stable operation in a high-concurrency environment.

[0004] The current spatiotemporal databases storing spatiotemporal data have difficulty achieving real-time distributed storage of tens of millions or hundreds of millions of data in the spatiotemporal dimension and real-time synchronous backup of multiple nodes. At the business level, it is also difficult to customize the storage of received data to designated spatial partitions.

[0005] The invention application with application number 202010836505.7 discloses a space-time gridded satellite data storage method and system. The satellite data storage method provided by the application is suitable for synchronous storage and processing of multiple satellites. On the one hand, multiple satellites have a consistent storage model, which is convenient for multi-satellite data networking storage and sharing services; on the other hand, it can directly and quickly perform storage and processing on a single satellite, thereby meeting the real-time requirements of remote sensing data for earth observation. However, it also has the problem that its method is only applicable to on-board storage, cannot be applied to large-scale space-time database management, does not adopt the message queue multi-node storage method, and cannot obtain reliable space-time data when the database is down or fails.

[0006] Based on the above reasons, it is necessary to develop a real-time distributed storage system and method for spatiotemporal data, adopt a multi-node synchronization method, so that spatiotemporal data can be safely and reliably stored in a distributed database for easy use. Summary of the invention

[0007] In view of the above-mentioned problems, the purpose of the present invention is to provide a distributed spatiotemporal database data synchronization system and method based on message queues, so as to realize distributed storage and synchronous backup of spatiotemporal data in the spatiotemporal dimension.

[0008] The embodiment of the present invention provides a distributed spatiotemporal database data synchronization system and method based on a message queue.

[0009] A first aspect: A distributed spatiotemporal database data synchronization system based on a message queue, the system comprising:

[0010] The producer identifies the spatial range of multi-source spatiotemporal data messages and delivers the spatiotemporal data messages to the spatial partition of the specified message queue based on the spatial range classification;

[0011] Message queue, based on spatial range classification to form spatial partitions to store spatiotemporal data messages;

[0012] The consumer group consumes the spatiotemporal data messages of the spatial partition of the consumption message queue, classifies the spatiotemporal data messages by time, and delivers the spatiotemporal data messages to the designated multi-node distributed database according to the spatiotemporal dimension characteristics;

[0013] Multi-node distributed database, multi-node distribution stores spatiotemporal data based on spatiotemporal dimension characteristics.

[0014] Second aspect:

[0015] The distributed spatiotemporal database data synchronization method based on the above system includes:

[0016] Obtain spatiotemporal data messages from multiple sources. The producer identifies the spatial range of the spatiotemporal data messages and delivers the spatiotemporal data messages to the spatial partition of the specified message queue based on the spatial range classification.

[0017] Use multiple consumer groups to pull spatially partitioned spatiotemporal data messages in the message queue, classify the spatiotemporal data messages by time, and store the spatiotemporal data messages in a multi-node distributed database with spatiotemporal dimension features.

[0018] Optionally, when the producer identifies the spatial range of the spatiotemporal data message, it uniformly cleans the multi-source spatiotemporal data messages and filters out the data in the specified spatial range.

[0019] Optionally, the message queue performs high-performance automated coordination and allocation of spatiotemporal data messages stored in spatial partitions to meet consumption needs of consumer groups.

[0020] Optionally, when the consumer group consumes the spatiotemporal data messages of the message queue, it handles problems arising in the message consumption process.

[0021] Optionally, multiple consumer groups consume the same spatiotemporal data message and send it to distributed spatiotemporal databases of different nodes for storage according to the spatiotemporal dimension.

[0022] Optionally, when a node distributed spatiotemporal database goes down or the fault is repaired, other node databases storing the same message re-submit the message to the consumer group to achieve rapid replenishment of database data and synchronous low-latency update of the faulty node database and other node database data.

[0023] A third aspect: An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the second aspect when executing the program.

[0024] A fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the second aspect.

[0025] Beneficial effects of the present invention:

[0026] 1. Based on the multi-topic subscription and broadcast consumption capabilities of the message queue, the present invention proposes a method for distributed storage and multi-node synchronous backup of spatiotemporal data in the spatiotemporal dimension. Messages are produced based on the spatiotemporal dimension and delivered to specific message queue topics, and data is pushed to multiple nodes based on broadcast consumption, thereby realizing distributed storage and synchronous backup of spatiotemporal data in the spatiotemporal dimension.

[0027] 2. The distributed spatiotemporal database of the present invention synchronizes data, deploys multiple consumer groups to consume the same message and stores it in databases of different nodes, to ensure that messages are not lost when encountering problems such as database downtime, and can achieve low-latency synchronous update of data at each node, providing basic multi-node support for data load query. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural schematic diagram of the distributed spatiotemporal database data synchronization system of the present invention;

[0029] Figure 2 It is a flow diagram of the distributed spatiotemporal database data synchronization method of the present invention;

[0030] Figure 3 This is a schematic diagram of the time and space partitioning of the message queue of the present invention;

[0031] Figure 4 This is a schematic diagram of the message group structure of the present invention;

[0032] Figure 5 This is a schematic diagram of the data structure of the distributed database of the present invention;

[0033] Figure 6 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0035] It is difficult to store data in the current spatiotemporal database, process tens of millions or hundreds of millions of records in real time, store data in a distributed manner in the spatiotemporal dimension, and perform real-time synchronous backup of multiple nodes. At the business level, it is also difficult to customize the storage of received data to designated spatial partitions.

[0036] In view of the above problems, the present invention provides a distributed spatiotemporal database data synchronization system based on message queues. Figure 1 A schematic diagram of the module structure of a distributed spatiotemporal database data synchronization system based on a message queue provided in an embodiment of the present invention, the system comprising: a producer, a message queue, a consumer group, a multi-node distributed database, etc.

[0037] Among them, the producer identifies the spatial range of the acquired multi-source spatiotemporal data messages, and uniformly cleans the spatiotemporal data according to the specified cleaning rules to obtain spatiotemporal data in a unified format, and then delivers the spatiotemporal data to the spatial partition specified by the message queue based on the spatial range classification.

[0038] The message queue is divided into spatial partitions based on the spatial range classification. The spatial partitions are used to store spatiotemporal data messages of different spatial ranges. For example Figure 3 As shown in the figure, the message queue space partitions are "dl_meta_topic", "items_topic", "meta_topic" and "real_topic". Each space partition can be divided by space coordinates to form different regional ranges.

[0039] When a consumer group consumes spatiotemporal data messages in a message queue, the message queue can perform high-performance automated coordination and allocation of spatiotemporal data messages stored in spatial partitions to meet the consumption needs of each consumer group and improve the consumption performance of the consumer group.

[0040] The consumer group is a program that receives messages and is composed of multiple consumers. The consumer group consumes messages quickly to achieve high-performance real-time processing of spatiotemporal data.

[0041] Consumer groups are divided into Figure 4As shown, for example: the consumer group includes "meta-group", "statistics-group" and "test-group". The "meta-group" consumer group includes "-text-5", "1", "2", "3" and "4". Consumers jointly perform high-performance real-time consumption processing of spatiotemporal data.

[0042] The consumer group consumes the spatiotemporal data messages of the spatial partition of the message queue, classifies the spatiotemporal data messages by time, and adds solutions to various problems that may arise during message processing. For example, when there are new requirements, such as changes in the time classification granularity, additions, deletions, and modifications to database table fields, the consumer group program can be modified to adapt to these problems. Improve the reliability of consumption and deliver spatiotemporal data messages to the specified multi-node distributed database according to the spatiotemporal dimension characteristics.

[0043] A multi-node distributed database is a database based on multi-node distribution. It can store spatiotemporal data with spatiotemporal dimension characteristics, and realize clustering and distributed storage effects of spatiotemporal data based on message queues and producer and consumer programs.

[0044] like Figure 5 The satellite spatiotemporal data shown is stored in a distributed database, including data such as satellite type, time parameters and spatial coordinates.

[0045] Furthermore, multiple consumer groups are used to consume the same spatiotemporal data message of the spatial partition of the message queue. The consumer groups deliver it to the distributed spatiotemporal database of different nodes for storage according to the spatiotemporal dimension. A spatiotemporal data message can be saved in the distributed spatiotemporal database of multiple nodes. When the distributed spatiotemporal database of one node goes down or the fault is repaired, the persistent storage characteristics of the message queue message are utilized, and the distributed spatiotemporal database of other nodes storing the same message is utilized to re-deliver the message to the consumer group, so as to realize rapid replenishment of database data and achieve synchronous low-latency update of database data with other nodes, providing basic multi-node support for load query of spatiotemporal data.

[0046] like Figure 2As shown, the present invention obtains multi-source spatiotemporal data messages, utilizes producers to identify the spatial range of spatiotemporal data messages, and delivers the spatiotemporal data messages to the spatial partition of the specified message queue based on spatial range classification; utilizes multiple consumer groups to pull the spatiotemporal data messages of the spatial partition in the message queue, performs time classification on the spatiotemporal data messages, and stores the spatiotemporal data messages in a multi-node distributed database with spatiotemporal dimension features; deploys multiple consumer groups to consume the same message and stores it in distributed databases of different nodes, and the distributed spatiotemporal database performs multi-node spatiotemporal data synchronization to ensure that message loss is prevented when encountering problems such as database downtime, and can achieve low-latency synchronous update of spatiotemporal data of each node, providing basic multi-node support for data load query.

[0047] The present invention also provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call the logic instructions in the memory, for example, to execute the following method:

[0048] Obtain spatiotemporal data messages from multiple sources. The producer identifies the spatial range of the spatiotemporal data messages and delivers the spatiotemporal data messages to the spatial partition of the specified message queue based on the spatial range classification. Use multiple consumer groups to pull the spatiotemporal data messages of the spatial partition in the message queue, classify the spatiotemporal data messages by time, and store the spatiotemporal data messages in a multi-node distributed database with spatiotemporal dimension features.

[0049] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0050] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0051] Obtain spatiotemporal data messages from multiple sources. The producer identifies the spatial range of the spatiotemporal data messages and delivers the spatiotemporal data messages to the spatial partition of the specified message queue based on the spatial range classification. Use multiple consumer groups to pull the spatiotemporal data messages of the spatial partition in the message queue, classify the spatiotemporal data messages by time, and store the spatiotemporal data messages in a multi-node distributed database with spatiotemporal dimension features.

[0052] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0053] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed spatiotemporal database data synchronization system based on message queue, characterized in that: The system comprises: The producer identifies the spatial range of multi-source spatiotemporal data messages and delivers the spatiotemporal data messages to the spatial partition of the specified message queue based on the spatial range classification; Message queue, based on spatial range classification to form spatial partitions to store spatiotemporal data messages; The consumer group consumes the spatiotemporal data messages of the spatial partition of the consumption message queue, classifies the spatiotemporal data messages by time, and delivers the spatiotemporal data messages to the designated multi-node distributed database according to the spatiotemporal dimension characteristics; Multi-node distributed database, multi-node distribution stores spatiotemporal data based on spatiotemporal dimension characteristics; Among them, multiple consumer groups consume the same spatiotemporal data message and send it to the distributed spatiotemporal database of different nodes for storage according to the spatiotemporal dimension.

2. A distributed spatiotemporal database data synchronization method based on the system of claim 1, characterized in that: include: Obtain spatiotemporal data messages from multiple sources. The producer identifies the spatial range of the spatiotemporal data messages and delivers the spatiotemporal data messages to the spatial partition of the specified message queue based on the spatial range classification. Use multiple consumer groups to pull spatially partitioned spatiotemporal data messages in the message queue, classify the spatiotemporal data messages by time, and store the spatiotemporal data messages in a multi-node distributed database with spatiotemporal dimension features.

3. The distributed spatiotemporal database data synchronization method according to claim 2, characterized in that: When the producer identifies the spatial range of the spatiotemporal data message, it uniformly cleans the multi-source spatiotemporal data messages and filters out the data in the specified spatial range.

4. The distributed spatiotemporal database data synchronization method according to claim 2, characterized in that: The message queue performs high-performance automated coordination and distribution of spatiotemporal data messages stored in spatial partitions to meet consumption needs of consumer groups.

5. The distributed spatiotemporal database data synchronization method according to claim 2, characterized in that: When the consumer group consumes the spatiotemporal data messages of the message queue, it handles problems that arise during the message consumption process.

6. The distributed spatiotemporal database data synchronization method according to claim 2, characterized in that: When a node distributed spatiotemporal database goes down or the fault is repaired, other node databases storing the same message will re-submit the message to the consumer group, so as to quickly replenish the database data and achieve synchronous low-latency update of the failed node database and other node database data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the distributed spatiotemporal database data synchronization method according to any one of claims 2 to 6 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed spatiotemporal database data synchronization method as described in any one of claims 2 to 6 are implemented.

Images