Fusion storage management system for multi-source image data

Abstract

The invention relates to a multi-source image data fusion storage management system in the field of data storage management systems, which comprises a system end and an equipment end, the equipment end comprises a data storage module, and the system end comprises a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module and a data unification module; the data storage module comprises a plurality of distributed storage areas, each distributed storage area comprises a distributed cache area and a file storage unit, the operation and maintenance module is used for monitoring a system load and dynamically adjusting a synchronization strategy of the front-end synchronization module according to the system load, and the level-to-level management module is used for setting priorities and association relationships of the distributed storage areas. According to the method, efficient data storage and management are achieved, the timeliness and stability of data synchronization are ensured through fine management of the access frequency, the capacity size and the system load condition of the data files, and meanwhile the utilization efficiency of system resources is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to an image data storage management system, in particular to a multi-source image data fusion storage management system applied to the field of data storage management systems. BACKGROUND

[0002] The existing multi-source image data storage management system aims to efficiently process image data from multiple sources. It has strong storage capacity and can accommodate massive amounts of image data in different formats, resolutions, and sources, such as satellite remote sensing, medical imaging, and images captured by surveillance cameras. In terms of management, the system uses indexing technology to quickly locate and retrieve specific images, improving data call efficiency. At the same time, through data classification and labeling mechanisms, images are classified according to their content, purpose, and other attributes, making it easy for users to find images by category. In addition, it has data backup and recovery functions to prevent data loss and ensure the security and integrity of image data, providing strong support for analysis and application based on multi-source image data in various industries.

[0003] Chinese patent CN117523418B discloses a multi-source remote sensing image analysis method and system. This invention is more comprehensive and effective in predicting and managing volatility and uncertainty in multi-source data. In addition, it improves the efficiency and effectiveness of processing different types of data sources. Furthermore, it can more deeply capture dynamic changes and complex behavior patterns among multi-source data. Finally, the application of spectral deconvolution-based remote sensing image enhancement technology effectively improves image quality.

[0004] Chinese patent CN118279167B discloses a mine layer management system based on multi-source data. This invention realizes the fusion of multi-source mine images by combining feature extraction, clustering analysis, coordinate reconstruction, and layer fusion technology, reducing the difficulty of multi-source data integration, improving data utilization, and thus improving management planning support.

[0005] The existing multi-source image data storage system lacks elastic synchronization strategies in high-load scenarios, leading to a decline in system performance. The existing image data storage system has problems such as insufficient dynamic resource scheduling, rigid synchronization strategies, and low cache management efficiency. SUMMARY

[0006] In view of the above existing technology, the technical problem to be solved by the present application is the problem of insufficient dynamic resource scheduling, rigid synchronization strategies, and low cache management efficiency in the existing image data storage system.

[0007] To solve the above problems, the application provides a multi-source image data fusion storage management system, which comprises a system end and a device end, the device end comprises a data storage module, and the system end comprises a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module and a data unification module; The data storage module comprises a plurality of distributed storage areas, the distributed storage area comprises a distributed cache area and a file storage unit, the distributed cache area is provided with a first cache space for periodic cleaning and a second cache space for long-term storage; The data acquisition module is used for connecting a plurality of image data sources and checking data integrity; The front-end synchronization module is used for data updating of the distributed storage area, and the front-end synchronization module executes a preset asynchronous synchronization strategy and a real-time synchronization strategy on cold data and hot data in the distributed storage area respectively; The operation and maintenance module is used for monitoring system load and dynamically adjusting the synchronization strategy of the front-end synchronization module according to the system load, adjusting the sharding transmission rate according to the system load for hot data greater than a set maximum capacity, executing batch submission and synchronizing the hot data to the distributed cache area for hot data less than a set minimum capacity, and dynamically adjusting the transmission interval and the sharding size according to the system load condition for medium-capacity hot data between the set maximum capacity and the set minimum capacity; The hierarchical management module is used for setting the priority and the association relationship of the distributed storage area, and when the distributed cache area space of the high-priority distributed storage area is insufficient, the second cache space of other distributed storage areas of the same level or lower priority associated with the high-priority distributed storage area is called to temporarily store data; The data unification module is used for format classification, format conversion and format unified management of data transmitted by the image data source.

[0008] As a further supplement to the application, when the second cache space temporarily stores data, the data files temporarily stored in the second cache space are synchronized to the original corresponding file storage unit, and then are transferred from the second cache space to the first cache space.

[0009] As a further supplement to the application, the hot data is a data file with a number of accesses greater than a set number within a set unit of time, and the cold data is a data file with a number of accesses less than a set threshold within a set unit of time.

[0010] As a further supplement to the application, the first cache space is cleaned during a system low-load period, and before cleaning the first cache space each time, the data file that is still determined as hot data beyond a set time is cached into the second cache space, and the data file in the second cache space is cleaned after being determined as cold data.

[0011] As a further supplement to the present application, the maximum capacity of the secondary cache space of the same priority called is 30%, and the maximum capacity of the secondary cache space of the low priority called is 50%.

[0012] As a further supplement to the present application, the asynchronous synchronization strategy is delayed synchronization. When the data source of the cold data is updated, the synchronization is delayed for a set time. If the system state is high load at the time point after the delay, the synchronization is continued to be delayed for a set time until the system load condition decreases to a set value.

[0013] As a further supplement to the present application, the distributed storage area is a plurality of data hard disks arranged in the hard disk cabinet. A storage controller for managing the plurality of data hard disks is arranged in the hard disk cabinet. The data hard disk comprises a bracket, and a hard disk main body and a control front end are mounted on the bracket. At least one associated interface is arranged at the side end of the control front end. A plurality of movable joints matched with the associated interfaces are arranged in the hard disk cabinet. The plurality of data hard disks are associated by being connected with the movable joints. An associated line matched with the plurality of movable joints is arranged in the hard disk cabinet.

[0014] As a further supplement to the present application, a local controller for controlling the movable joint is mounted on the hard disk cabinet. The control of the movable joint is performed by the local controller, and remote operation is prohibited.

[0015] As a further supplement to the present application, the associated interface is provided with two, and the data hard disk connected with the two associated interfaces is provided with a high-priority distributed storage area.

[0016] As a further supplement to the present application, a mainboard for connecting with the plurality of data hard disks is arranged in the hard disk cabinet. A slot matched with the data hard disk is formed on the hard disk cabinet. A limiting sliding slot matched with the associated interface is formed on the side wall of the slot opening. The movable joint comprises a connecting column sliding in the limiting sliding slot. An electromagnet is mounted at the bottom end of the limiting sliding slot. A flexible data line is connected between the electromagnet and the connecting column. The flexible data line is connected with the mainboard through the associated line embedded in the hard disk cabinet.

[0017] In summary, the present application realizes efficient data storage and management. Through fine management of the access frequency, capacity size and system load condition of the data file, the timeliness and stability of data synchronization are ensured, and the utilization efficiency of system resources is improved. Through the hierarchical management function, the system can flexibly cope with the cache demand of different priority storage areas, and realizes the optimized storage and rapid access of data. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The storage management system block diagram of the first embodiment of the present application; Figure 2Logical flow chart for data updating of the storage management system of the first embodiment of the present application; Figure 3 Working block diagram for working of the storage management system of the first embodiment of the present application; Figure 4 Perspective view of the hard disk cabinet of the second embodiment of the present application; Figure 5 Perspective view of the hard disk cabinet when the data hard disk is extracted of the second embodiment of the present application; Figure 6 Top view of the data hard disk of the second embodiment of the present application; Figure 7 For Figure 6 Structural schematic diagram at A in the middle.

[0019] Explanation of the figure numbers: 1, hard disk cabinet; 2, data hard disk; 21, bracket; 22, hard disk main body; 23, front end control; 24, movable joint. DETAILED DESCRIPTION

[0020] The two embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0021] First embodiment: Figures 1-3 It is shown that the fusion storage management system of multi-source image data includes a system end and a device end, the device end includes a data storage module, and the system end includes a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module and a data unification module; The data acquisition module is used to connect multiple image data sources and check data integrity; The data storage module includes multiple distributed storage areas, and the distributed storage area includes a distributed cache area and a file storage unit; The front-end synchronization module is used for data updating of the distributed storage area, and the front-end synchronization module executes different synchronization strategies for cold data and hot data; hot data adopts a preset real-time synchronization strategy when the system is under a set load condition (such as CPU utilization < 30%); hot data is a data file with a number of accesses greater than a set number within a set unit of time, for example, a data with an access frequency ≥ 100 times / day; Cold data is a data file with a number of accesses less than a set threshold within a set unit of time, for example, a data with an access frequency ≤ 1 time / week; cold data is preset to adopt an asynchronous synchronization strategy; the asynchronous synchronization strategy is a delayed synchronization; When the data source of the cold data is updated, the synchronization is performed with a delay set time, if the system state is high load at the time point after the delay, the synchronization is continued to be performed with a delay set time until the system load condition is reduced to a set value, the delay time is set by the person skilled in the art according to the prior art, for example, the delay time is dynamically increased with the system load, and the initial delay is 1 hour.

[0022] The distributed cache area is provided with a first cache space for periodic cleaning and a second cache space for long-term storage, and the data in the first cache space and the second cache space can be migrated to each other; The operation and maintenance module is used for monitoring the system load and dynamically adjusting the synchronization strategy of the front-end synchronization module according to the system load, the system load is divided into three grades according to the CPU utilization, low load: CPU utilization < 40%; medium load: 40% ≥ CPU utilization ≥ 80%; high load: CPU utilization > 80%; The above delay synchronization strategy is maintained for the cold data; For hot data greater than the set maximum capacity (such as long video, data greater than 1 GB), the shard transmission rate is adjusted according to the system load, and the shard transmission rate adjustment method is set by the person skilled in the art according to the prior art, for example, determined by the following formula: shard transmission rate = reference bandwidth × (1-load rate), shard size = load rate × 1 GB; The shard transmission rate can also be set to be reduced at high load, for example, the shard transmission rate is reduced by 50% when the CPU utilization is ≥ 80%; For hot data less than the set minimum capacity (such as small capacity pictures, data less than 100 MB), the hot data file performs batch submission and synchronizes it to the distributed cache area, and a single batch ≥ 50 hot data files; the corresponding file storage unit of the distributed cache area asynchronously reads the updated storage data, realizing the update of the corresponding file in the file storage unit; For medium-capacity hot data (such as short video or high-definition picture, data between 100 MB-1 GB) between the set maximum capacity and the set minimum capacity, the transmission interval and the shard size are dynamically adjusted according to the system load condition, and the specific adjustment method is set by the person skilled in the art; For example, when the system is low load, the transmission interval of the medium-capacity hot data is set to 5s, and no shard transmission is performed; When the system is medium load, the transmission interval of the medium-capacity hot data = 5s × (load coefficient), and the shard size = 500MB × (load coefficient); When the system is high load, the transmission interval of the medium-capacity hot data is set to ≥ 10s, and the shard size is ≤ 200MB; The load coefficient = current CPU utilization / reference utilization, and the reference utilization is defined by the person skilled in the art, for example, 40%; The hierarchical management module is used for setting priorities and association relationships of the distributed storage areas. When the cache space of a distributed storage area with high priority is insufficient, the secondary cache space of other distributed storage areas with the same level or lower priority associated with the distributed storage area is called to temporarily store data, so as to realize branch temporary storage of data. When the secondary cache space temporarily stores data, the data files temporarily stored in the secondary cache space are synchronized to the file storage units corresponding to the data files, and then the data files are transferred from the secondary cache space to the primary cache space. The primary cache space is cleaned up during a low-load period of the system. The cleaning interval is greater than the set unit time for hot data judgment. Before the primary cache space is cleaned up each time, the data files that are still judged as hot data after exceeding the set time are called into the secondary cache space. The data files in the secondary cache space are cleaned up after being judged as cold data.

[0023] The data uniform module is used for converting data transmitted by multiple image data sources into a uniform format and then transmitting the data to corresponding distributed storage areas.

[0024] The maximum capacity of the secondary cache space with the same priority called is 30%, and the maximum capacity of the secondary cache space with a low priority called is 50%.

[0025] In the embodiment, data files are divided into hot data and cold data according to access frequencies of the data files. For hot data files frequently accessed, the system adopts a real-time synchronization strategy to ensure instant updating of data. For cold data files with low access frequencies, the system adopts an asynchronous synchronization strategy to reduce occupation of system resources. When a data source of cold data is updated, the system delays synchronization for a period of time. If the system is in a high-load state at this time, the system continues to delay until the system load is reduced to a proper level to perform synchronization.

[0026] The system of the embodiment further performs further data synchronization management according to sizes of data capacities and system load conditions. For hot data greater than a set maximum capacity, such as long videos, the system dynamically adjusts a sharding transmission rate according to system load to ensure stability and efficiency of data transmission. For hot data smaller than a set minimum capacity, such as small pictures, the system adopts a batch submission mode to synchronize the data to a distributed cache area to improve synchronization efficiency.

[0027] For medium-capacity hot data between the set maximum capacity and the set minimum capacity, the system dynamically adjusts a transmission interval and a sharding size according to system load conditions. For example, when the system is in a low-load state, the transmission interval is short, and no sharding transmission is performed. When the system is in a medium-load state, the transmission interval and the sharding size are dynamically adjusted according to a load coefficient. When the system is in a high-load state, the transmission interval is lengthened, and the sharding size is reduced to reduce occupation of system resources.

[0028] The system of the embodiment also has a hierarchical management function, and can perform data temporary storage and cleaning according to the priorities and association relationships of the distributed storage areas; when the cache space of a high-priority distributed storage area is insufficient, the system can call the secondary cache space of other distributed storage areas associated with the high-priority distributed storage area and having a lower priority to temporarily store data, and then transfer the data to the primary cache space after the data is synchronized to the original corresponding file storage unit; at the same time, the system can also clean the primary cache space during a low-load period of the system to ensure the effective use of the cache space.

[0029] The scheme realizes efficient data storage and management, ensures the timeliness and stability of data synchronization, and improves the utilization efficiency of system resources through fine management of the access frequency, capacity size and system load of data files; through the hierarchical management function, the system can flexibly cope with the cache requirements of distributed storage areas of different priorities, and realizes optimized storage and fast access of data.

[0030] The second embodiment: Figures 4-7 It is shown that, compared with the first embodiment, the present embodiment selects a plurality of data hard disks 2 arranged in the hard disk cabinet 1 as the distributed storage area; the hard disk cabinet 1 is connected with the front-end synchronization module, and the front-end synchronization module is used to transmit the data updated by the image data source to the specified data hard disk 2 in the hard disk cabinet 1; The distributed storage area is a plurality of data hard disks 2 arranged in the hard disk cabinet 1, and the hard disk cabinet 1 is provided with a storage controller for managing the plurality of data hard disks 2; the storage controller adopts an existing technology, and is used to centrally manage the distributed cache areas of the plurality of data hard disks 2, and supports multi-disk cooperative reading and writing; The data hard disk 2 comprises a bracket 21, a hard disk main body 22 and a control front end 23 are arranged on the bracket 21, at least one association interface is arranged at the side end of the control front end 23, a plurality of movable joints 24 matched with the association interfaces are arranged in the hard disk cabinet 1, the plurality of data hard disks 2 are associated by being connected with the movable joints 24, and the hard disk cabinet 1 is provided with an association line matched with the plurality of movable joints 24; the data hard disks 2 are connected with each other through the association line and the movable joint 24 to establish a data connection; the data hard disks 2 read the distributed cache areas of the associated data hard disks 2 through the association line; A mainboard for connecting with the plurality of data hard disks 2 is arranged in the hard disk cabinet 1, and a slot matched with the data hard disk 2 is formed in the hard disk cabinet 1; a limiting sliding groove matched with the association interface is formed in the side wall of the slot opening, the movable joint 24 comprises a connecting column sliding in the limiting sliding groove, an electromagnet is arranged at the bottom end of the limiting sliding groove, a flexible data line is connected between the electromagnet and the connecting column, and the flexible data line is connected with the mainboard through the association line embedded in the hard disk cabinet 1.

[0031] The hard disk cabinet 1 is provided with a local controller for regulating the movable joint 24, the regulation of the movable joint 24 is performed by the local controller, the remote operation is prohibited, the regulation of the movable joint 24 is prohibited, the risk of illegal access and data leakage is effectively prevented, and after the adjustment of the movable joint 24, the local controller uploads the adjusted associated relationship to the hierarchical management module, and the priority of each data hard disk 2 associated is selected and set at the hierarchical management module.

[0032] The hierarchical management module sets the priority to the storage controller, and the storage controller can identify the associated relationship between the data hard disks 2. After the data hard disks 2 are associated, when the data hard disks 2 with high priority perform large-capacity data updating, the system end can temporarily store the updated large-capacity data in the distributed cache area of another data hard disk 2; optionally, the associated interface is set to two, and the data hard disks 2 connected with the two associated interfaces are set as high-priority distributed storage areas; it is convenient to set the data hard disks 2 with high priority through the local control; During data synchronization, the data synchronization is performed according to the synchronization task queue, when the data required to be synchronized by the data hard disk 2 is temporarily stored in another data hard disk 2, the storage controller controls the data hard disk 2 performing data synchronization to read the distributed cache area of the corresponding data hard disk 2 to obtain the temporarily stored data in the distributed cache area; The embodiment realizes flexible association and efficient data synchronization between the plurality of data hard disks 2 in the hard disk cabinet 1; through the set associated line, it is convenient to adjust the associated relationship of the plurality of data hard disks 2, and it is convenient to cooperatively process during data synchronization; when data is updated, the system can temporarily store the large-capacity data in the distributed cache area of another data hard disk 2 during large-capacity data synchronization, on the one hand, it is convenient to update the large-capacity data to the system in time, and on the other hand, it is easy to avoid the performance decline of a single data hard disk 2 due to a large amount of data updating; The storage controller can sequentially perform data synchronization according to the synchronization task queue, and ensures the consistency and integrity of the data.

[0033] In combination with the current actual demand, the protection scope of the above-mentioned embodiments of the present application is not limited thereto, various changes made within the knowledge range of those skilled in the art without departing from the concept of the present application still fall within the protection scope of the present application.

Claims

1. A fusion storage management system of multi-source image data, comprising a system end and a device end, the device end comprising a data storage module, and the system end comprising a data acquisition module, a front-end synchronization module, an operation and maintenance module, a hierarchical management module and a data unification module; characterized in that: the data storage module comprises a plurality of distributed storage areas, the distributed storage area comprises a distributed cache area and a file storage unit, the distributed cache area is provided with a first cache space for periodic cleaning and a second cache space for long-term storage; the data acquisition module is used to connect a plurality of image data sources and check data integrity; the front-end synchronization module is used for data update of the distributed storage area, and the front-end synchronization module executes a preset asynchronous synchronization strategy and a real-time synchronization strategy on cold data and hot data in the distributed storage area, respectively; the operation and maintenance module is used to monitor system load and dynamically adjust the synchronization strategy of the front-end synchronization module according to the system load, for hot data greater than a set maximum capacity, to adjust the sharding transmission rate according to the system load; for hot data less than a set minimum capacity, to perform batch submission and synchronize it to the distributed cache area; for medium-capacity hot data between the set maximum capacity and the set minimum capacity, to dynamically adjust the transmission interval and the sharding size according to the system load condition; the hierarchical management module is used to set the priority and association relationship of the distributed storage area, when the distributed cache area of the high-priority distributed storage area is insufficient, to call the second cache space of other distributed storage areas of the same level or lower priority associated therewith for data temporary storage; the data unification module is used for format classification, format conversion and format unified management of data transmitted by the image data source; when the second cache space temporarily stores data, the data file temporarily stored in the second cache space is synchronized to the original corresponding file storage unit, and then it is transferred from the second cache space to the first cache space; the hot data is a data file with a number of accesses greater than a set number in a set unit of time, and the cold data is a data file with a number of accesses less than a set threshold in a set unit of time; the first cache space is cleaned during a system low-load period, before each first cache space cleaning, data files that have been judged as hot data for more than a set time are cached into the second cache space, and data files in the second cache space are judged as cold data and then cleaned; the maximum capacity of the second cache space of the same priority called is 30%, and the maximum capacity of the second cache space of the lower priority called is 50%; the asynchronous synchronization strategy is delayed synchronization, when the data source of the cold data is updated, the synchronization is delayed for a set time, if the system state is high load at the time point after the delay, the synchronization is continued to be delayed for a set time, until the system load condition decreases to a set value. ​ ​ ​ ​ ​ ​ 2. The system for fused storage management of multi-source image data according to claim 1, characterized in that: ​ 3. The system for fused storage management of multi-source image data according to claim 1, characterized in that: ​ 4. The system for fused storage management of multi-source image data of claim 1, wherein: ​ 5. The system for fused storage management of multi-source image data of claim 1, wherein: ​ 6. The system for fused storage management of multi-source image data of claim 1, wherein: ​ 7. The system for fused storage management of multi-source image data of claim 1, wherein: The distributed storage area is a plurality of data hard disks (2) arranged in a hard disk cabinet (1), the hard disk cabinet (1) is provided with a storage controller for managing the plurality of data hard disks (2), the data hard disk (2) comprises a bracket (21), a hard disk main body (22) and a regulation front end (23) are installed on the bracket (21), at least one associated interface is arranged at the side end of the regulation front end (23), a plurality of movable joints (24) matched with the associated interface are arranged in the hard disk cabinet (1), the plurality of data hard disks (2) are associated by being connected with the movable joints (24), and the hard disk cabinet (1) is provided with associated lines matched with the plurality of movable joints (24).

8. The system for fused storage management of multi-source image data according to claim 7, characterized in that: The hard disk cabinet (1) is provided with a local controller for regulating the movable joint (24), the regulation of the movable joint (24) is performed by the local controller, and remote operation is prohibited.

9. The system for fused storage management of multi-source image data of claim 7, wherein: The associated interface is provided with two, and the data hard disk (2) connected with the two associated interfaces is provided as a high-priority distributed storage area.

10. The system for fused storage management of multi-source image data of claim 7, wherein: The hard disk cabinet (1) is provided with a mainboard connected with the plurality of data hard disks (2), the hard disk cabinet (1) is provided with a slot matched with the data hard disk (2), a limiting sliding groove matched with the associated interface is arranged on the side wall of the slot opening, the movable joint (24) comprises a connecting column sliding in the limiting sliding groove, an electromagnet is arranged at the bottom end of the limiting sliding groove, a flexible data line is connected between the electromagnet and the connecting column, and the flexible data line is connected with the mainboard through the associated lines embedded in the hard disk cabinet (1).

Images