Data distributed storage method and device

Abstract

This disclosure provides a distributed data storage method and apparatus, relating to the field of distributed database technology, and particularly to the fields of distributed data storage, spatial data storage, remote sensing image processing, map surveying, and geographic information systems. One specific implementation of the method includes: uniquely mapping each spatial unit of a target spatial region to a corresponding logical shard; establishing a time-ordered binding relationship between each logical shard and a storage node, recording the effective time of each binding relationship, and generating a logical shard-storage node mapping table sorted by effective time; when performing data writing, determining the corresponding logical shard based on the spatial unit to which the data to be written belongs, and matching storage nodes falling within the corresponding effective time interval in the logical shard-storage node mapping table based on the generation time of the data to be written, and writing the data to the matching storage node, thereby achieving high throughput, high balance, and migration-free data writing.

Description

Technical Field

[0001] This disclosure relates to the field of distributed database technology, and in particular to the fields of distributed data storage, spatial data storage, remote sensing image processing, map surveying, and geographic information systems. Background Technology

[0002] In large-scale remote sensing image recognition and processing scenarios, after the model completes processes such as building extraction and ground feature identification, massive amounts of structured or semi-structured recognition result data are continuously generated. This type of data is generated in parallel by distributed recognition tasks and has characteristics such as large data scale, high write concurrency, and strong correlation with spatial regions. Related storage and management technologies have become an important direction for industry research and application.

[0003] Distributed storage technology is widely used in large-scale data storage, with mainstream implementations including consistent hashing-based data distribution mechanisms and cluster-based data sharding schemes. Consistent hashing maps data to different storage nodes by constructing hash rings, optimizing data migration overhead and improving system scalability during node scaling. Cluster architecture distributes data across multiple nodes using predefined sharding rules and slot allocation strategies, achieving system load balancing and efficient parallel access. While these distributed storage solutions have achieved good results in general data storage scenarios, they are prone to mismatches between data distribution and spatial characteristics in applications such as remote sensing image recognition data, leading to decreased locality of access. Furthermore, under high-concurrency write conditions, uneven node load and write hotspots can easily occur. Summary of the Invention

[0004] This disclosure presents a distributed data storage method and apparatus.

[0005] In a first aspect, embodiments of this disclosure propose a distributed data storage method, comprising: spatially dividing a target spatial region to obtain multiple spatial units; uniquely mapping each spatial unit to a corresponding logical shard; establishing a time-ordered binding relationship between each logical shard and a storage node, and recording the effective time corresponding to each binding relationship, generating a logical shard-storage node mapping table sorted by effective time; when performing data writing, determining the corresponding logical shard according to the spatial unit to which the written data belongs, and matching the storage node falling within the corresponding effective time interval in the logical shard-storage node mapping table according to the generation time of the written data, and writing the written data to the matching storage node.

[0006] Secondly, embodiments of this disclosure propose a distributed data storage device, comprising: a partitioning module configured to spatially partition a target spatial region to obtain multiple spatial units; a mapping module configured to uniquely map each spatial unit to a corresponding logical shard; a generation module configured to establish a time-ordered binding relationship between each logical shard and a storage node, and record the effective time corresponding to each binding relationship, generating a logical shard-storage node mapping table sorted by effective time; and a writing module configured to, when performing data writing, determine the corresponding logical shard according to the spatial unit to which the written data belongs, and match the storage node falling within the corresponding effective time interval in the logical shard-storage node mapping table according to the generation time of the written data, and write the written data to the matching storage node.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described in the first aspect.

[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] The key or essential features of the embodiments disclosed herein are not intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein: Figure 1 This is a flowchart of an embodiment of the data distributed storage method according to the present disclosure; Figure 2 This is a flowchart of yet another embodiment of the data distributed storage method according to the present disclosure; Figure 3 This is a schematic diagram of data writing routing based on street segmentation and time binding; Figure 4 This is a flowchart of a time-ordered binding-based data migration and expansion method; Figure 5 This is a schematic diagram of the structure of an embodiment of a distributed data storage device according to the present disclosure; Figure 6 This is a block diagram of an electronic device used to implement the data distributed storage method of the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0014] Figure 1 A flow 100 of an embodiment of a distributed data storage method according to the present disclosure is shown. The distributed data storage method includes the following steps: Step 101: Divide the target spatial region into multiple spatial units.

[0015] In this embodiment, the entity executing the distributed data storage method can spatially divide the target spatial region into multiple spatial units. The purpose of spatial division is to break down a continuous geographical region into non-overlapping, fully covered minimum management units, giving data a clear spatial affiliation and improving spatial locality and query efficiency.

[0016] In some embodiments, the target spatial area is spatially divided according to the road network to obtain multiple street block units. Using road edges as natural boundaries to divide streets allows identified targets such as buildings, land features, and roads to belong to a single street block, avoiding data ownership conflicts caused by crossing boundaries, and is suitable for the storage and management of remote sensing image recognition results.

[0017] In some embodiments, the target spatial region is spatially divided into multiple spatial units using regular grid partitioning, administrative region partitioning, or area of ​​interest (AOI) partitioning. Regular grid partitioning divides the target spatial region into regular rectangular grid units according to preset latitude and longitude intervals or pixel sizes, ensuring uniform unit size and regular distribution. Administrative region partitioning completes the spatial division according to administrative boundaries such as provinces, cities, districts, and streets, facilitating matching and integration with government geographic information data. Area of ​​Interest (AOI) partitioning involves custom boundary division for key monitored and identified geographic areas, focusing on core identification areas and improving the targeting of data storage and retrieval. All of the above spatial partitioning methods can achieve standardized partitioning of the target region, meeting the spatial unit management needs of different remote sensing image recognition scenarios. Step 102: Each spatial unit is uniquely mapped to its corresponding logical slice.

[0018] In this embodiment, the aforementioned execution entity can uniquely map each spatial unit to a corresponding logical slice. Each spatial unit corresponds to a fixed logical slice, and one logical slice can correspond to multiple spatial units. The mapping relationship remains unchanged during system operation, expansion, and node changes, providing a foundation for migration-free expansion.

[0019] In some embodiments, the mapping steps between spatial units and logical segments can be as follows: First, assign a unique spatial identifier to each spatial unit. This spatial identifier can be, for example, a street number, geocode, or grid identifier, used to uniquely identify the spatial unit.

[0020] Next, a hash calculation is performed on the spatial identifier of the spatial unit to obtain the hash value corresponding to the spatial unit. The hash algorithm converts the spatial identifier into a uniformly distributed value, avoiding data distribution skew.

[0021] Then, the hash value is moduloed based on the number of logical shards to obtain the remainder. For example, if the total number of shards is 1024, the hash value modulo 1024 will yield a remainder between 0 and 1023.

[0022] Finally, each spatial unit is uniquely mapped to a logical partition whose partition number matches the remainder. For example, if the remainder is 66, the spatial unit is permanently mapped to partition 66, achieving a stable, uniform, and predictable data distribution.

[0023] The mapping method described above allows data from each spatial unit to be evenly distributed across different logical shards, avoiding hotspot issues caused by centralized data writing. Simultaneously, a fixed mapping relationship is established between spatial units and logical shards, eliminating the need to recalculate, adjust, or migrate data when the system expands or nodes change. This effectively reduces system overhead and improves the stability, write throughput, and scalability of distributed storage.

[0024] In some embodiments, consistent hashing, range sharding, or spatial grid partitioning are used to map spatial units to corresponding logical shards. Consistent hashing achieves a stable mapping between spatial units and logical shards by constructing hash rings, adjusting only a small amount of data distribution when the number of shards changes, thus improving system scalability. Range sharding divides shards according to continuous numerical intervals of spatial identifiers, ensuring that spatially adjacent units belong to the same or adjacent shards, enhancing spatial query locality. Spatial grid partitioning includes quadtree partitioning and geographic grid partitioning. Quadtree partitioning uses a recursive hierarchical grid structure to adapt to multi-scale spatial data. Geographic grid partitioning uses standard geographic grid coding to achieve unique spatial location identification. All of the above methods can achieve efficient and stable sharding mapping results.

[0025] Step 103: Establish a time-ordered binding relationship between each logical shard and the storage node, record the effective time of each binding relationship, and generate a logical shard-storage node mapping table sorted by effective time.

[0026] In this embodiment, the aforementioned execution entity can establish a time-ordered binding relationship between each logical shard and a storage node, and record the effective time corresponding to each binding relationship, generating a logical shard-storage node mapping table sorted by effective time. The storage node can be a database instance, or it can employ a distributed key-value storage system, object storage system, or hybrid storage architecture to meet the needs of different application scenarios. A logical shard can be bound to multiple storage nodes, and each binding record contains an effective time; the mapping table is arranged in ascending order of time. For example, shard 5 is bound to nodes DB1 (2026-01-01), DB2 (2026-04-01), and DB3 (2026-07-01) sequentially, forming a time-ordered routing rule.

[0027] Generally, the earlier the effective time, the earlier the corresponding storage node will accept the recognition task data generated earlier; the newer the effective time, the newer the data written will be accepted, thus achieving natural distribution by time and avoiding data migration during expansion.

[0028] In some embodiments, the binding order between each logical shard and storage node is determined using version numbers, time interval identifiers, or task sequence numbers. The version number identifies the update iteration order of the binding relationship; a larger version number indicates a newer binding relationship. The time interval identifier directly limits the data write time range corresponding to each storage node, enabling rapid matching of storage nodes for the target time period. The task sequence number determines the routing order of storage nodes according to the generation sequence number of the remote sensing image recognition task. All of these methods can represent the binding relationship between logical shards and storage nodes with an ordered sequence, achieving a routing effect consistent with time-ordered binding without relying on specific effective timestamps, further enhancing the adaptability of this solution under different storage architectures and business scenarios.

[0029] Step 104: When performing data writing, determine the corresponding logical shard based on the spatial unit to which the data to be written belongs, and match the storage node that falls within the corresponding effective time interval in the logical shard-storage node mapping table according to the generation time of the data to be written, and write the data to the matching storage node.

[0030] In this embodiment, when the aforementioned execution entity performs data writing, it determines the corresponding logical shard based on the spatial unit to which the data to be written belongs, and matches the storage node that falls within the corresponding effective time interval in the logical shard-storage node mapping table based on the generation time of the data to be written, and writes the data to the matching storage node.

[0031] The write process consists of two steps: first, logical shards are located using spatial units; then, time-matched storage nodes are generated to achieve dual-dimensional routing based on both space and time. This matching process employs a range-based hit mechanism, which allows for rapid location of storage nodes once the time range is determined, eliminating the need to traverse all binding relationships and further improving write routing efficiency.

[0032] In some embodiments, based on the creation time of the remote sensing image recognition task, storage nodes falling within the corresponding effective time interval are matched in the logical sharding-storage node mapping table, and the recognition results corresponding to the remote sensing image recognition task are written to the matched storage nodes. The recognition results include structured spatial data such as building outlines, land feature classification, road extraction, and change detection. The task creation time is used as the data generation time for routing and writing.

[0033] The distributed data storage method provided in this disclosure achieves high throughput, high balance, and migration-free data writing through spatial fixed mapping and time-ordered binding, and is particularly suitable for spatial data storage such as remote sensing image recognition results.

[0034] Figure 2A flow 200 of another embodiment of the distributed data storage method according to this disclosure is shown. The distributed data storage method includes the following steps: Step 201: Divide the target spatial region into multiple spatial units.

[0035] Step 202: Uniquely map each spatial unit to its corresponding logical slice.

[0036] Step 203: Establish a time-ordered binding relationship between each logical shard and a storage node, record the effective time of each binding relationship, and generate a logical shard-storage node mapping table sorted by effective time.

[0037] Step 204: When performing data writing, determine the corresponding logical shard based on the spatial unit to which the data to be written belongs, and match the storage node that falls within the corresponding effective time interval in the logical shard-storage node mapping table according to the generation time of the data to be written, and write the data to the matching storage node.

[0038] In this embodiment, the specific operations of steps 201-204 can be referred to Figure 1 The relevant descriptions of steps 101-104 in the corresponding embodiments will not be repeated here.

[0039] Step 205: In response to the detection that the load of the storage node has reached the preset expansion threshold, add a new storage node to the corresponding logical shard, establish a new binding relationship between the new storage node and the corresponding logical shard, and record the effective time of the new binding relationship.

[0040] In this embodiment, in response to the detection that the load of a storage node has reached a preset expansion threshold, the aforementioned execution entity can add a new storage node to the corresponding logical shard, establish a new binding relationship between the new storage node and the corresponding logical shard, and record the effective time of the new binding relationship. The monitoring metrics include storage utilization, write QPS (Queries Per Second), CPU (Central Processing Unit) load, and disk I / O (Input / Output). The preset expansion threshold can be flexibly configured according to business scenarios, such as when storage utilization reaches 80%, write QPS exceeds 10,000, or CPU load consistently exceeds 70%, ensuring that the system can complete elastic expansion before high load. Expansion only adds nodes and a new effective time; there is no migration, redistribution, or service interruption, achieving completely seamless online expansion. New data is automatically written to the new node, while historical data is retained on the original node.

[0041] Step 206: When performing a data query, determine the corresponding logical shard based on the spatial unit to which the query data belongs, and match the storage nodes that fall within the corresponding effective time interval in the logical shard-storage node mapping table according to the time range of the query data, and read the query data from the matched storage nodes.

[0042] In this embodiment, when the aforementioned execution entity performs a data query, it determines the corresponding logical shard based on the spatial unit to which the query data belongs, and matches storage nodes falling within the corresponding effective time interval in the logical shard-storage node mapping table according to the time range of the query data, and reads the query data from the matched storage nodes. The query supports spatial range, time range, and spatiotemporal joint queries. The system first locates the shard, then filters nodes by time, reads the data in parallel, and merges the results, significantly improving query efficiency.

[0043] For batch queries across time intervals, parallel read requests can be initiated to multiple target storage nodes simultaneously, significantly reducing overall query latency and adapting to the rapid retrieval needs of large amounts of remote sensing image data.

[0044] The distributed data storage method provided in this disclosure combines spatial fixed mapping, temporal ordered routing, and a migration-free expansion mechanism to effectively support high-concurrency writing and dynamic expansion requirements, achieving high-concurrency writing, migration-free expansion, and efficient spatiotemporal querying, thus fully meeting the large-scale storage requirements of remote sensing image recognition results.

[0045] Addressing the characteristics of large-scale, high-concurrency, and strong spatial attributes in remote sensing image recognition data, this method enables efficient data distribution and stable access in high-concurrency write scenarios. It also supports online expansion of the storage system without interrupting recognition operations, thereby improving the overall system stability and scalability. This method is applicable to remote sensing image intelligent interpretation platforms, geographic information system platforms, spatial data processing systems, and various artificial intelligence analysis platforms, supporting typical business scenarios such as building extraction, road recognition, land feature classification, and change detection. Through this solution, system expansion can be achieved without data migration while ensuring continuous and stable writing for large-scale parallel recognition tasks, improving data processing efficiency and reducing the impact on business continuity. It can be widely applied in fields such as smart cities, natural resource management, and mapping.

[0046] Compared with existing solutions, the distributed storage method disclosed herein can bring the following benefits in practical applications: 1. Improve system throughput and eliminate write bottlenecks: Based on the storage strategy of logical sharding and time-ordered binding, the recognition result data can be evenly distributed to multiple storage nodes, effectively alleviating the write pressure of a single point, avoiding hot node bottlenecks, and significantly improving the overall write throughput.

[0047] 2. Enables seamless online expansion and reduces system overhead: By adding time binding relationships, new and old data are naturally separated. The expansion process does not require moving or redistributing historical data, which greatly reduces the performance loss and maintenance costs caused by expansion.

[0048] 3. Ensure uninterrupted business operation and improve system stability: The expansion process does not require pausing identification tasks or data writing, and can be completed smoothly while the business is running normally, improving system availability and continuous service capabilities.

[0049] 4. Simplify operation and maintenance complexity and improve project maintainability: By decoupling the spatial sharding strategy and node expansion logic, the system can automatically and smoothly expand as the data scale grows, reducing manual intervention and complex configuration operations.

[0050] 5. Enhanced scenario adaptability and better suited to spatial data characteristics: Mechanism optimization is performed to address the strong spatial correlation and high-concurrency write characteristics of remote sensing identification results, resulting in superior performance compared to general distributed storage solutions in terms of spatial locality, data organization efficiency, and query performance.

[0051] Figure 3 A schematic diagram of data write routing based on block segmentation and time binding is shown.

[0052] The data writing process is based on blocks as the basic unit: First, the remote sensing image recognition task is divided according to blocks, with each recognition task corresponding to one block, and the task creation time is recorded; then, the block ID is hashed and a modulo operation is performed based on the preset number of blocks N to obtain the logical block ID, thereby realizing a unique and fixed mapping from spatial unit to logical block.

[0053] The sharding routing layer maintains a time-ordered binding relationship between each logical shard and a storage node, forming an ordered mapping list of shard ID → (storage node DB, effective time T). During the routing decision phase, based on the creation time of the identification task, the effective time interval within the shard binding relationship is matched. For example, when the task time satisfies "street binding time T1 ≤ task time < street binding time T2", the data is routed to DB1; when the task time satisfies "task time ≥ street binding time T2", it is routed to DB2. Finally, the identification result data is written to the matched storage node for storage.

[0054] This process achieves automatic data write distribution through a two-layer routing mechanism that binds spatial unit hash sharding and time order, providing a foundation for subsequent online expansion without data migration.

[0055] Figure 4 A flowchart of a time-ordered binding-based data-free migration and expansion method is shown.

[0056] After the process begins, remote sensing image recognition tasks are first generated at the street level, and the creation time of the tasks is recorded. Then, a hash operation is performed based on the street identifier, and modulo is taken based on the preset number of segments to obtain the segment ID corresponding to the street. Next, the list of storage node binding relationships under the segment, sorted by effective time, is queried. After that, a time matching judgment is performed based on the creation time of the recognition task. If an old effective time interval is matched, the data is written to the old storage node (old DB); if a new effective time interval is matched, the data is written to the new storage node (new DB).

[0057] After the data is written, the system will determine whether expansion is needed. If the determination is "no", the process ends directly; if the determination is "yes", expansion operation is performed: a new storage node is added, a new binding relationship is established for the corresponding shard and the new effective time T_new is recorded. Subsequent generated recognition tasks will be automatically written to the new storage node according to their creation time, realizing smooth expansion without data migration, and finally the process ends.

[0058] This process integrates data writing and dynamic expansion, enabling elastic expansion of system capacity without migrating historical data or interrupting the identification task, effectively ensuring system stability and scalability in high-concurrency writing scenarios.

[0059] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a distributed data storage device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0060] like Figure 5 As shown, the distributed data storage device 500 of this embodiment may include: a partitioning module 501, a mapping module 502, a generation module 503, and a writing module 504. The partitioning module 501 is configured to spatially partition the target spatial region to obtain multiple spatial units; the mapping module 502 is configured to uniquely map each spatial unit to a corresponding logical shard; the generation module 503 is configured to establish a time-ordered binding relationship between each logical shard and a storage node, record the effective time corresponding to each binding relationship, and generate a logical shard-storage node mapping table sorted by effective time; the writing module 504 is configured to, when performing data writing, determine the corresponding logical shard based on the spatial unit to which the written data belongs, and, based on the generation time of the written data, match storage nodes falling within the corresponding effective time interval in the logical shard-storage node mapping table, and write the written data to the matching storage node.

[0061] In this embodiment, the specific processing of the partitioning module 501, mapping module 502, generation module 503, and writing module 504 in the data distributed storage device 500, and the resulting technical effects, can be found in references to [the relevant documentation]. Figure 1 The relevant descriptions of steps 101-104 in the corresponding embodiments will not be repeated here.

[0062] In some optional implementations of this embodiment, the mapping module 502 is further configured to: assign a unique spatial identifier to the spatial unit; perform a hash calculation on the spatial identifier of the spatial unit to obtain the hash value corresponding to the spatial unit; perform a modulo operation on the hash value based on the number of logical shards to obtain the remainder corresponding to the hash value; and uniquely map the spatial unit to a logical shard whose shard number is consistent with the remainder.

[0063] In some optional implementations of this embodiment, the mapping module 502 is further configured to: map spatial units to corresponding logical partitions using a consistent hashing algorithm, a range partitioning method, or a spatial grid partitioning method, wherein the spatial grid partitioning includes quadtree partitioning and geographic grid partitioning. In some optional implementations of this embodiment, the partitioning module 501 is further configured to: spatially partition the target spatial area according to the road network to obtain multiple street block units.

[0064] In some optional implementations of this embodiment, the partitioning module 501 is further configured to: spatially divide the target spatial region using a regular grid partitioning, administrative region partitioning, or interest-based region partitioning to obtain multiple spatial units.

[0065] In some optional implementations of this embodiment, the generation module 503 is further configured to determine the order of binding relationships between each logical shard and the storage node by using a version number, time interval identifier, or task sequence number.

[0066] In some optional implementations of this embodiment, the writing module 504 is further configured to: match the storage node that falls within the corresponding effective time interval in the logical sharding-storage node mapping table according to the creation time of the remote sensing image recognition task, and write the recognition result corresponding to the remote sensing image recognition task into the matching storage node.

[0067] In some optional implementations of this embodiment, the distributed data storage device 500 further includes: an expansion module, configured to add a new storage node to the corresponding logical shard in response to detecting that the load of the storage node has reached a preset expansion threshold; establish a new binding relationship between the new storage node and the corresponding logical shard, and record the effective time of the new binding relationship.

[0068] In some optional implementations of this embodiment, the distributed data storage device 500 further includes: a query module, configured to determine the corresponding logical shard according to the spatial unit to which the query data belongs when performing a data query, and to match the storage node falling into the corresponding effective time interval in the logical shard-storage node mapping table according to the time range of the query data, and read the query data from the matched storage node.

[0069] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0070] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0071] In some optional implementations of this embodiment, Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0072] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0073] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the data distributed storage method. For example, in some embodiments, the data distributed storage method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data distributed storage method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data distributed storage method by any other suitable means (e.g., by means of firmware).

[0075] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0076] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0077] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0078] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0079] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0080] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0081] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0082] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A distributed data storage method, comprising: The target spatial region is spatially divided to obtain multiple spatial units; Each spatial unit is uniquely mapped to its corresponding logical slice; Establish a time-ordered binding relationship between each logical shard and a storage node, record the effective time corresponding to each binding relationship, and generate a logical shard-storage node mapping table sorted by effective time; When performing data writing, the corresponding logical shard is determined according to the spatial unit to which the data to be written belongs, and the storage node that falls within the corresponding effective time interval is matched in the logical shard-storage node mapping table according to the generation time of the data to be written, and the data to be written is written to the matching storage node.

2. The method according to claim 1, wherein, The step of uniquely mapping each spatial unit to its corresponding logical slice includes: Assign a unique spatial identifier to the spatial unit; Perform a hash calculation on the spatial identifier of the spatial unit to obtain the hash value corresponding to the spatial unit; The remainder corresponding to the hash value is obtained by taking the modulo of the number of logical shards. The spatial unit is uniquely mapped to a logical segment whose segment number matches the remainder.

3. The method according to claim 1, wherein, The step of uniquely mapping each spatial unit to its corresponding logical slice includes: The spatial units are mapped to corresponding logical partitions using a consistent hashing algorithm, range partitioning, or spatial grid partitioning. The spatial grid partitioning includes quadtree partitioning and geographic grid partitioning.

4. The method according to any one of claims 1-3, wherein, The target spatial region is spatially divided into multiple spatial units, including: The target spatial area is spatially divided according to the road network to obtain multiple street block units.

5. The method according to any one of claims 1-3, wherein, The target spatial region is spatially divided into multiple spatial units, including: The target spatial region is spatially divided into multiple spatial units by using regular grid division, administrative region division, or interest-based region division.

6. The method according to any one of claims 1-3, wherein, The step of establishing a time-ordered binding relationship between each logical shard and a storage node includes: The order in which the binding relationship between each logical shard and the storage node is determined by using version number, time interval identifier, or task sequence number.

7. The method according to claim 4, wherein, The step of matching a storage node that falls within the corresponding effective time interval in the logical shard-storage node mapping table based on the generation time of the written data, and writing the written data to the matching storage node, includes: Based on the creation time of the remote sensing image recognition task, a storage node that falls within the corresponding effective time interval is matched in the logical sharding-storage node mapping table, and the recognition result corresponding to the remote sensing image recognition task is written into the matched storage node.

8. The method according to any one of claims 1-7, wherein, The method further includes: In response to the detection that the load of a storage node has reached a preset expansion threshold, a new storage node is added to the corresponding logical shard. Establish a new binding relationship between the new storage node and the corresponding logical shard, and record the effective time of the new binding relationship.

9. The method according to any one of claims 1-8, wherein, The method further includes: When performing a data query, the corresponding logical shard is determined according to the spatial unit to which the query data belongs, and the storage node that falls within the corresponding effective time interval is matched in the logical shard-storage node mapping table according to the time range of the query data, and the query data is read from the matched storage node.

10. A distributed data storage device, comprising: The partitioning module is configured to spatially divide the target spatial region into multiple spatial units; The mapping module is configured to uniquely map each spatial unit to a corresponding logical slice. The generation module is configured to establish a time-ordered binding relationship between each logical shard and a storage node, record the effective time of each binding relationship, and generate a logical shard-storage node mapping table sorted by effective time. The write module is configured to determine the corresponding logical shard based on the spatial unit to which the write data belongs when performing data writing, and to match the storage node that falls within the corresponding effective time interval in the logical shard-storage node mapping table according to the generation time of the write data, and write the write data to the matching storage node.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images