Fusion storage management system of multi-source image data

Abstract

The application relates to a multi-source image data fusion storage management system in the field of data storage management systems, which comprises a system end and a device end; the device end comprises a data storage module; the system end comprises a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module and a data unification module; the data storage module comprises a plurality of distributed storage areas; the distributed storage area comprises a distributed cache area and a file storage unit; the operation and maintenance module is used for monitoring system load and dynamically adjusting the synchronization strategy of the front-end synchronization module according to the system load; and the hierarchical management module is used for setting the priority and the association relationship of the distributed storage area, so that efficient data storage and management are realized; through fine management of the access frequency, the capacity size and the system load condition of the data file, the timeliness and stability of data synchronization are ensured, and the utilization efficiency of system resources is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to an image data storage management system, in particular to a multi-source image data fusion storage management system applied to the field of data storage management systems. BACKGROUND

[0002] The existing multi-source image data storage management system aims to efficiently process image data from multiple sources. It has strong storage capacity and can accommodate massive amounts of image data in different formats, resolutions, and sources, such as satellite remote sensing, medical images, and images captured by surveillance cameras. In terms of management, the system uses indexing technology to quickly locate and retrieve specific images, improving data call efficiency. At the same time, through data classification and labeling mechanisms, images are classified according to their content, purpose, and other attributes, making it easy for users to find images by category. In addition, it has data backup and recovery functions to prevent data loss and ensure the security and integrity of image data, providing strong support for analysis and application based on multi-source image data in various industries.

[0003] Chinese patent CN117523418B discloses a multi-source remote sensing image analysis method and system. This invention is more comprehensive and effective in predicting and managing volatility and uncertainty in multi-source data. In addition, it improves the efficiency and effectiveness of processing different types of data sources. At the same time, this invention can more deeply capture dynamic changes and complex behavior patterns between multi-source data. Finally, the application of spectral deconvolution-based remote sensing image enhancement technology effectively improves the quality of images.

[0004] Chinese patent CN118279167B discloses a mine layer management system based on multi-source data. This invention realizes the fusion of multi-source mine images by combining feature extraction, clustering analysis, coordinate reconstruction, and layer fusion technology, reducing the difficulty of multi-source data integration, improving data utilization, and thus improving management planning support.

[0005] The existing multi-source image data storage system lacks elastic synchronization strategies in high-load scenarios, leading to a decline in system performance. The existing image data storage system has problems such as insufficient dynamic resource scheduling, rigid synchronization strategies, and low cache management efficiency. SUMMARY

[0006] In view of the above prior art, the technical problem to be solved by the present application is the problem of insufficient dynamic resource scheduling, rigid synchronization strategies, and low cache management efficiency in the existing image data storage system.

[0007] To solve the above problems, the application provides a multi-source image data fusion storage management system, which comprises a system end and a device end, the device end comprises a data storage module, and the system end comprises a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module and a data unification module;

[0008] The data storage module comprises a plurality of distributed storage areas, the distributed storage area comprises a distributed cache area and a file storage unit, the distributed cache area is provided with a first cache space for periodic cleaning and a second cache space for long-term storage;

[0009] The data acquisition module is used for connecting a plurality of image data sources and checking data integrity;

[0010] The front-end synchronization module is used for data updating of the distributed storage area, and the front-end synchronization module executes a preset asynchronous synchronization strategy and a real-time synchronization strategy on cold data and hot data in the distributed storage area respectively;

[0011] The operation and maintenance module is used for monitoring system load and dynamically adjusting the synchronization strategy of the front-end synchronization module according to the system load, for hot data greater than a set maximum capacity, adjusting the sharding transmission rate according to the system load; for hot data less than a set minimum capacity, executing batch submission and synchronizing to the distributed cache area; for medium-capacity hot data between the set maximum capacity and the set minimum capacity, dynamically adjusting the transmission interval and the sharding size according to the system load condition;

[0012] The hierarchical management module is used for setting the priority and association relationship of the distributed storage area, when the distributed cache area space of the high-priority distributed storage area is insufficient, calling the second cache space of other distributed storage areas of the same level or lower priority associated therewith for data temporary storage;

[0013] The data unification module is used for format classification, format conversion and format unified management of data transmitted by the image data source.

[0014] As a further supplement of the application, when the second cache space temporarily stores data, the data files temporarily stored in the second cache space are synchronized to the original corresponding file storage unit, and then transferred from the second cache space to the first cache space.

[0015] As a further supplement of the application, the hot data is a data file with a number of accesses greater than a set number in a set unit of time, and the cold data is a data file with a number of accesses less than a set threshold in a set unit of time.

[0016] As a further supplement of the present application, the primary cache space is cleaned up during the system low load period, and before each primary cache space cleanup, the data file cache that is still determined as hot data beyond the set time is called into the secondary cache space, and the data file in the secondary cache space is determined as cold data and is cleaned up.

[0017] As a further supplement of the present application, the maximum capacity of the secondary cache space of the same priority called is 30%, and the maximum capacity of the secondary cache space of the low priority called is 50%.

[0018] As a further supplement of the present application, the asynchronous synchronization strategy is delayed synchronization, and when the data source of the cold data is updated, the synchronization is delayed for a set time, and if the system state at the time point after the delay is high load, the synchronization is continued to be delayed for a set time until the system load condition is reduced to a set value.

[0019] As a further supplement of the present application, the distributed storage area is a plurality of data hard disks arranged in the hard disk cabinet, a storage controller for managing the plurality of data hard disks is arranged in the hard disk cabinet, the data hard disk comprises a bracket, a hard disk main body and a control front end are installed on the bracket, at least one association interface is arranged at the side end of the control front end, a plurality of movable joints matched with the association interface are arranged in the hard disk cabinet, the plurality of data hard disks are associated by being connected with the movable joints, and an association line matched with the plurality of movable joints is arranged in the hard disk cabinet.

[0020] As a further supplement of the present application, a local controller for controlling the movable joint is installed on the hard disk cabinet, the control of the movable joint is performed by the local controller, and remote operation is prohibited.

[0021] As a further supplement of the present application, the association interface is provided with two, and the data hard disks connected with the two association interfaces are provided as the high-priority distributed storage area.

[0022] As a further supplement of the present application, a mainboard for connecting with the plurality of data hard disks is arranged in the hard disk cabinet, a slot matched with the data hard disk is formed on the hard disk cabinet, a limiting sliding slot matched with the association interface is formed on the side wall of the slot opening, the movable joint comprises a connecting column sliding in the limiting sliding slot, an electromagnet is installed at the bottom end of the limiting sliding slot, a flexible data line is connected between the electromagnet and the connecting column, and the flexible data line is connected with the mainboard through the association line embedded in the hard disk cabinet.

[0023] In summary, the present application realizes efficient data storage and management, ensures the timeliness and stability of data synchronization through fine management of the access frequency, capacity size and system load condition of the data file, and improves the utilization efficiency of system resources; through the hierarchical management function, the system can flexibly cope with the cache demand of different priority storage areas, and realizes the optimized storage and rapid access of data. Attached Figure Description

[0024] Figure 1 This is a block diagram of the storage management system according to the first embodiment of this application;

[0025] Figure 2 A logic flowchart of the storage management system in the first embodiment of this application when updating data;

[0026] Figure 3 A block diagram illustrating the operation of the storage management system according to the first embodiment of this application;

[0027] Figure 4 This is a perspective view of the hard disk enclosure according to the second embodiment of this application;

[0028] Figure 5 A perspective view of the data hard drive being removed from the hard drive enclosure according to the second embodiment of this application;

[0029] Figure 6 This is a top cross-sectional view of the data hard disk according to the second embodiment of this application;

[0030] Figure 7 for Figure 6 A schematic diagram of the structure at point A in the middle.

[0031] Explanation of the labels in the diagram:

[0032] 1. Hard drive enclosure; 2. Data hard drive; 21. Bracket; 22. Hard drive body; 23. Control front end; 24. Connector. Detailed Implementation

[0033] The two embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0034] Implementation method 1:

[0035] Figures 1-3 The diagram shows a multi-source image data fusion storage and management system, which includes a system end and a device end. The device end includes a data storage module, and the system end includes a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module, and a data unification module.

[0036] The data acquisition module is used to connect to multiple image data sources and verify data integrity.

[0037] The data storage module includes multiple distributed storage areas, which in turn include distributed cache areas and file storage units;

[0038] The front-end synchronization module is used for data update of the distributed storage area, and the front-end synchronization module executes different synchronization strategies for cold data and hot data respectively; the hot data adopts a preset real-time synchronization strategy when the system is in a set load condition (such as CPU utilization < 30%); the hot data is a data file with a visit frequency greater than a set number of times in a set unit of time, for example, a visit frequency ≥ 100 times / day;

[0039] The cold data is a data file with a visit frequency less than a set threshold in a set unit of time, for example, a visit frequency ≤ 1 time / week; the cold data adopts a preset asynchronous synchronization strategy; the asynchronous synchronization strategy is a delayed synchronization;

[0040] When the data source of the cold data is updated, the synchronization is performed after a delay of a set time; if the system state is high load at the time point after the delay, the synchronization is continued after a delay of a set time, until the system load condition decreases to a set value; the delay time is set by a person skilled in the art according to the prior art, for example, the delay time is dynamically increased with the system load, and the initial delay is 1 hour.

[0041] The distributed cache area is provided with a first cache space for periodic cleaning and a second cache space for long-term storage; the data in the first cache space and the second cache space can be migrated to each other;

[0042] The operation and maintenance module is used for monitoring the system load and dynamically adjusting the synchronization strategy of the front-end synchronization module according to the system load; the system load is divided into three grades according to the system CPU utilization, low load: CPU utilization < 40%; medium load: 40% ≥ CPU utilization ≥ 80%; high load: CPU utilization > 80%;

[0043] The above delayed synchronization strategy is maintained for the cold data;

[0044] For hot data greater than a set maximum capacity (such as a long video with a data amount greater than 1 GB), the sharding transmission rate is adjusted according to the system load; the sharding transmission rate adjustment mode is set by a person skilled in the art according to the prior art, for example, determined by the following formula: sharding transmission rate = reference bandwidth × (1-load rate), and the sharding size = load rate × 1 GB;

[0045] The sharding transmission rate can also be set to be reduced at high load, for example, the sharding transmission rate is reduced by 50% when the CPU utilization is ≥ 80%;

[0046] For hot data less than a set minimum capacity (such as a small capacity picture with a data amount less than 100 MB), the hot data file executes batch submission and is synchronized to the distributed cache area; a single batch ≥ 50 hot data files; the corresponding file storage unit of the distributed cache area asynchronously reads the updated storage data, to realize the update of the corresponding file in the file storage unit;

[0047] For medium-capacity hot data (such as short videos or high-definition pictures, data volume between 100MB-1GB) between the set maximum capacity and the set minimum capacity, the transmission interval and the slice size are dynamically adjusted according to the system load condition, and the specific adjustment method is set by a person skilled in the art;

[0048] For example, when the system is low in load, the transmission interval of medium-capacity hot data is set to 5s, and no slicing transmission is performed;

[0049] When the system is medium in load, the transmission interval of medium-capacity hot data = 5s x (load coefficient), and the slice size = 500MB x (load coefficient);

[0050] When the system is high in load, the transmission interval of medium-capacity hot data is set to ≥10s, and the slice size is ≤200MB;

[0051] The load coefficient = current CPU utilization rate / reference utilization rate, and the reference utilization rate is defined by a person skilled in the art, such as 40%;

[0052] The hierarchical management module is used to set the priority and association relationship of the distributed storage area. When the cache space of the distributed storage area with high priority is insufficient, the secondary cache space of other distributed storage areas with the same level or lower priority associated with it is called to temporarily store data, realizing data storage in branches. When the secondary cache space temporarily stores data, the data files temporarily stored in the secondary cache space are synchronized to the original corresponding file storage unit, and then they are transferred from the secondary cache space to the primary cache space;

[0053] The primary cache space is cleaned during the low-load period of the system, and the cleaning interval period is greater than the set unit time of hot data judgment. Before each primary cache space cleaning, the data file cache that exceeds the set time and is still judged as hot data is called into the secondary cache space. The data files in the secondary cache space are judged as cold data and are cleaned.

[0054] The data unification module is used to convert the data transmitted by multiple image data sources into a unified format and then transmit it to the corresponding distributed storage area.

[0055] The maximum capacity of the secondary cache space with the same priority called is 30%, and the maximum capacity of the secondary cache space with low priority called is 50%.

[0056] In this embodiment, the data files are divided into hot data and cold data according to the access frequency. For hot data files that are frequently accessed, the system adopts a real-time synchronization strategy to ensure the instant update of data. For cold data files with low access frequency, the system adopts an asynchronous synchronization strategy to reduce the occupation of system resources. When the data source of cold data is updated, the system delays synchronization for a period of time. If the system is in a high load state at this time, the delay continues until the system load decreases to an appropriate level before synchronization.

[0057] The system of this embodiment also performs further data synchronization management according to the capacity size of data and the system load condition. For hot data larger than the set maximum capacity, such as long videos, the system dynamically adjusts the sharding transmission rate according to the system load to ensure the stability and efficiency of data transmission. For hot data smaller than the set minimum capacity, such as small pictures, the system uses batch submission to synchronize them to the distributed cache area to improve synchronization efficiency.

[0058] For medium-capacity hot data between the set maximum capacity and the set minimum capacity, the system dynamically adjusts the transmission interval and the sharding size according to the system load condition. For example, when the system is in a low load state, the transmission interval is short and no sharding transmission is performed. When the system is in a medium load state, the transmission interval and the sharding size are dynamically adjusted according to the load coefficient. When the system is in a high load state, the transmission interval is lengthened and the sharding size is reduced to reduce the occupation of system resources.

[0059] The system of this embodiment also has a hierarchical management function that can perform data temporary storage and cleaning according to the priority and association of the distributed storage areas. When the cache space of a high-priority distributed storage area is insufficient, the system calls the secondary cache space of other distributed storage areas associated with it and with lower priority to temporarily store data. After the data is synchronized to the original corresponding file storage unit, it is transferred to the primary cache space. At the same time, the system also cleans the primary cache space during the low load period of the system to ensure the effective use of cache space.

[0060] This scheme realizes efficient data storage and management. Through fine management of the access frequency, capacity size, and system load condition of data files, the timeliness and stability of data synchronization are ensured, and the utilization efficiency of system resources is also improved. Through the hierarchical management function, the system can flexibly respond to the cache needs of storage areas with different priorities, realizing the optimized storage and fast access of data.

[0061] Second embodiment:

[0062] Figures 4-7As shown, compared with the first embodiment, the present embodiment selects the plurality of data hard disks 2 arranged in the hard disk cabinet 1 as the distributed storage area; the hard disk cabinet 1 is connected with the front-end synchronization module, and the data updated by the image data source is transmitted to the specified data hard disk 2 in the hard disk cabinet 1 through the front-end synchronization module;

[0063] The distributed storage area is the plurality of data hard disks 2 arranged in the hard disk cabinet 1, and the hard disk cabinet 1 is provided with a storage controller for managing the plurality of data hard disks 2. The storage controller adopts the prior art, and the storage controller is used for centrally managing the distributed cache areas of the plurality of data hard disks 2, which supports multi-disk cooperative reading and writing.

[0064] The data hard disk 2 comprises a bracket 21, a hard disk main body 22 and a control front end 23 are arranged on the bracket 21, at least one association interface is arranged at the side end of the control front end 23, a plurality of movable joints 24 matched with the association interface are arranged in the hard disk cabinet 1, the plurality of data hard disks 2 are associated by being connected with the movable joints 24, and the hard disk cabinet 1 is provided with an association line matched with the plurality of movable joints 24, and the data hard disks 2 associated with each other establish data connection through the association line and the movable joint 24; the data hard disk 2 reads the distributed cache area of the associated data hard disk 2 through the association line.

[0065] The hard disk cabinet 1 is provided with a mainboard for connecting with the plurality of data hard disks 2, the hard disk cabinet 1 is provided with a slot matched with the data hard disk 2, a limiting sliding groove matched with the association interface is arranged on the side wall of the slot opening, the movable joint 24 comprises a connecting column sliding in the limiting sliding groove, an electromagnet is arranged at the bottom end of the limiting sliding groove, a flexible data line is connected between the electromagnet and the connecting column, and the flexible data line is connected with the mainboard through the association line embedded in the hard disk cabinet 1.

[0066] The hard disk cabinet 1 is provided with a local controller for controlling the movable joint 24, the control of the movable joint 24 is performed by the local controller, and remote operation is prohibited. The control of the movable joint 24 is prohibited, the risk of illegal access and data leakage is effectively prevented, the local controller uploads the adjusted association relationship after the movable joint 24 is adjusted to the hierarchical management module, and the priority of each data hard disk 2 associated is selected and arranged at the hierarchical management module.

[0067] The hierarchical management module synchronizes the information to the storage controller after setting the priority, and the storage controller can identify the association relationship between the data hard disks 2;

[0068] After the data hard disks 2 are associated, when the data hard disk 2 with high priority performs large-capacity data updating, the system end can temporarily store the updated large-capacity data in the distributed cache area of another data hard disk 2; optionally, the association interface is provided with two, and the data hard disks 2 connected with the two association interfaces are provided as the high-priority distributed storage area; it is convenient to quickly set the data hard disk 2 with high priority through the local control.

[0069] During data synchronization, the data required to be synchronized by the data hard disk 2 is temporarily stored in other data hard disks 2, and the storage controller controls the data hard disk 2 that is currently performing data synchronization to read the distribution cache area of the corresponding data hard disk 2 to obtain the data temporarily stored in the distribution cache area.

[0070] The embodiment realizes flexible association and efficient data synchronization among the plurality of data hard disks 2 in the hard disk cabinet 1. Through the set association line, the association relationship of the plurality of data hard disks 2 is facilitated to be adjusted, and the collaborative processing during data synchronization is facilitated. When data is updated, the system can temporarily store the large-capacity data in the distribution cache area of other data hard disks 2 during large-capacity data synchronization, on the one hand, facilitating the timely update of the large-capacity data to the system, and on the other hand, facilitating the performance decline of a single data hard disk 2 caused by a large amount of data update to be avoided.

[0071] The storage controller can sequentially perform data synchronization according to the synchronization task queue, ensuring the consistency and integrity of the data.

[0072] In combination with the current actual demand, the above-mentioned embodiments adopted by the present application are not limited to this, various changes made within the knowledge range of those skilled in the art without departing from the concept of the present application still fall within the protection range of the present application.

Claims

1. A fusion storage and management system for multi-source image data, comprising a system end and a device end, wherein the device end includes a data storage module, and the system end includes a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module, and a data unification module; characterized in that: The data storage module includes multiple distributed storage areas, each of which includes a distributed cache area and a file storage unit. The distributed cache area is equipped with a primary cache space that is periodically cleaned up and a secondary cache space for long-term storage. The data acquisition module is used to connect to multiple image data sources and verify data integrity. The front-end synchronization module is used for data updates in the distributed storage area. The front-end synchronization module executes preset asynchronous synchronization strategies and real-time synchronization strategies for cold data and hot data in the distributed storage area, respectively. The operation and maintenance module is used to monitor the system load and dynamically adjust the synchronization strategy of the front-end synchronization module according to the system load. For hot data with a capacity greater than the set maximum capacity, the fragmentation transmission rate is adjusted according to the system load. For hot data with a capacity less than the set minimum capacity, batch submission is performed and the data is synchronized to the distributed buffer. For medium-capacity hot data with a capacity between the set maximum capacity and the set minimum capacity, the transmission interval and fragmentation size are dynamically adjusted according to the system load. The hierarchical management module is used to set the priority and association of distributed storage areas. When the distributed cache space of a high-priority distributed storage area is insufficient, the secondary cache space of other distributed storage areas of the same or lower priority associated with it is called to temporarily store the data. The data unification module is used for format classification, format conversion, and unified format management of data transmitted from image data sources; The asynchronous synchronization strategy is delayed synchronization. When the data source of cold data is updated, synchronization is performed after a set delay. If the system is under high load at the delayed time, synchronization is performed again after a set delay until the system load drops to the set value.

2. The system for fused storage management of multi-source image data according to claim 1, characterized in that: When the secondary cache space temporarily stores data, the data file temporarily stored in the secondary cache space will be transferred from the secondary cache space to the primary cache space after being synchronized to its original corresponding file storage unit.

3. The system for fused storage management of multi-source image data according to claim 1, characterized in that: The hot data is a data file that has been accessed more than a set number of times within a set unit of time, and the cold data is a data file that has been accessed less than a set threshold within a set unit of time.

4. The system for fused storage management of multi-source image data of claim 1, wherein: The first-level cache space is cleaned up during periods of low system load. Before each cleanup of the first-level cache space, data files that are still considered hot data after a set time are cached and moved to the second-level cache space. Data files in the second-level cache space are cleaned up after being identified as cold data.

5. The system for fused storage management of multi-source image data of claim 1, wherein: The maximum capacity of the L2 cache space that can be accessed at the same priority is 30%, and the maximum capacity of the L2 cache space that can be accessed at the lower priority is 50%.

6. The system for fused storage management of multi-source image data of claim 1, wherein: The distributed storage area consists of multiple data hard disks (2) set in a hard disk enclosure (1). The hard disk enclosure (1) is equipped with a storage controller for managing the multiple data hard disks (2). Each data hard disk (2) includes a bracket (21). The bracket (21) is equipped with a hard disk body (22) and a control front end (23). The control front end (23) has at least one associated interface on its side. The hard disk enclosure (1) is equipped with multiple movable connectors (24) that match the associated interfaces. The multiple data hard disks (2) are associated by connecting with the movable connectors (24). The hard disk enclosure (1) is equipped with associated lines that match the multiple movable connectors (24).

7. The system for fused storage management of multi-source image data according to claim 6, characterized in that: The hard disk enclosure (1) is equipped with a local controller for regulating the movable connector (24). The regulation of the movable connector (24) is performed by the local controller, and remote operation is prohibited.

8. The system for fused storage management of multi-source image data according to claim 6, wherein: The associated interface is set to two, and the data hard disk (2) that is connected to the two associated interfaces is set to a high-priority distributed storage area.

9. The system for fused storage management of multi-source image data of claim 6, wherein: The hard disk enclosure (1) is equipped with a motherboard for connecting to multiple data hard disks (2). The hard disk enclosure (1) has slots that match the data hard disks (2). The side wall of the slot opening has a limiting groove that matches the associated interface. The movable connector (24) includes a docking post that slides in the limiting groove. An electromagnet is installed at the bottom of the limiting groove. A flexible data cable is connected between the electromagnet and the docking post. The flexible data cable is connected to the motherboard through an associated line embedded in the hard disk enclosure (1).

Images