A mixed approach for monitoring platforms that uses unified mounting of multiple storage disk types and dynamic task allocation.

Abstract

This invention discloses a method for unified mounting and dynamic task allocation of multiple types of storage disks in a monitoring platform, belonging to the field of video surveillance technology. The method abstracts the storage resources into virtual disks for unified management by the monitoring platform; and includes the detected virtual disks into a writable disk pool. When a new or restartable recording task occurs on the monitoring platform, a task scheduling cycle is triggered. The used space increase rate and current remaining available space of the disks in the writable disk pool are statistically analyzed, and a candidate disk set with low load and high remaining space is selected. A disk is then randomly selected from this set to allocate tasks. This invention achieves mixed use and load balancing of multiple types of storage disks. A failure of one type of storage medium does not affect recording tasks on other types of media, improving the overall reliability of the platform. Simultaneously, users can flexibly choose storage media according to their needs, expanding the selection range and reducing additional procurement costs. It is suitable for recording storage scenarios in various video surveillance platforms.

Description

Technical Field

[0001] This invention belongs to the field of video surveillance technology, specifically involving a mixed-use method for dynamically allocating dynamic tasks while uniformly mounting multiple types of storage disks on a surveillance platform. Background Technology

[0002] In the field of video surveillance technology, video recording and storage is one of the core functions of a video surveillance platform. The video data generated by the surveillance will eventually be persistently stored in various storage media in the form of files. The selection and management of storage media directly affects the stable operation and data security of the video surveillance platform.

[0003] With the diversified development of video surveillance services, users of different platforms have different needs for storage media due to significant differences in business scenarios, cost budgets, and performance requirements. Some users may prefer to use local disks with lower costs, some users may need flexible NFS cloud storage, and some users may choose object storage or private protocol storage due to data security or special functional requirements. There is even a need to use a combination of different types of storage media (i.e., "mixed use") to adapt to the actual needs of complex business scenarios.

[0004] Currently, existing technical solutions for video recording storage in the video surveillance platform field generally have a mandatory requirement for "consistent storage media type," which can be implemented in two ways: First, it requires platform users to use only the same type of storage media, with all recording tasks stored uniformly in this single type of media, and the platform only developing corresponding management modules for this type of storage media; Second, it requires users to use storage media corresponding to specific hardware devices pre-bound to the platform, and to achieve storage connection through platform-specific drivers, without allowing users to choose third-party storage media or non-bound storage media.

[0005] The existing technical solutions for video recording and storage in the aforementioned video surveillance platforms have significant drawbacks in practical applications, as detailed below:

[0006] 1. Low platform storage reliability and single point of failure risk: Existing solutions mandate the use of the same type of storage media or storage media bound to specific hardware, making all recording tasks on the platform dependent on a single type of storage media. When problems occur with the hardware environment (e.g., local disk failure, cloud storage server hardware failure) or network environment (e.g., NFS cloud storage network interruption, private storage connection anomalies), the lack of alternative storage media will directly lead to the interruption of all recording tasks on the entire platform, failing to guarantee continuous storage of recorded data and severely impacting the normal operation of video surveillance services.

[0007] 2. Limited storage options for users and increased costs: On the one hand, existing solutions restrict users to using only specified types of storage media or storage devices bound to specific hardware. If a user already has storage media of a different type (e.g., object storage has been deployed, but the platform only supports local disks), these existing storage media cannot be used on the platform, resulting in wasted resources. On the other hand, to meet the platform's storage requirements, users need to purchase additional storage media of specified types or storage devices bound to specific hardware. This not only increases the user's initial hardware investment costs but also fails to meet the user's actual need to flexibly choose storage media based on their own business needs (e.g., cost control, performance pursuit).

[0008] While the existing technical solutions can achieve basic video recording and storage, they cannot meet users' needs for flexible selection and mixed use of multiple types of storage media. They also have the reliability risk of "a single storage media failure causing the entire platform to be interrupted in recording." Therefore, there is an urgent need for a technical solution suitable for video surveillance platforms to achieve effective management and mixed use of multiple types of storage disks, and to solve the problems of insufficient storage flexibility, poor compatibility and low reliability in the existing solutions. Summary of the Invention

[0009] In view of the shortcomings of existing technologies, this invention provides a mixed-use method for unified mounting and dynamic task allocation of multiple types of storage disks in a monitoring platform. It is applicable to scenarios involving the unified mounting, dynamic task allocation, and mixed use of four types of storage media—local disks, NFS cloud storage, object storage, and private storage—in video surveillance platforms. The aim is to improve the efficiency of recording task allocation and the overall reliability of the storage system through storage management logic. Specifically, the invention aims to: 1) enhance the overall reliability of the video surveillance platform's storage system by supporting the mixed use of multiple types of storage disks, avoiding the interruption of recording tasks across the entire platform due to hardware failures or network anomalies of a single type of storage medium. This ensures that even if one type of storage medium malfunctions, recording tasks on other types of storage media can still execute normally, guaranteeing continuous storage of recording data. 2) expand the range of storage media choices for video surveillance platform users, breaking the limitations of "using only specified types of storage media" or "binding to specific hardware storage media." It supports users connecting local disks, NFS cloud storage, object storage, and private storage to the platform and using them in combination, allowing users to flexibly select storage media based on business scenarios (such as cost control and performance requirements), reducing additional procurement costs and avoiding waste of existing storage resources.

[0010] To achieve the above objectives, the technical solution adopted by this invention is: a method for the unified mounting of multiple types of storage disks and dynamic task allocation in a monitoring platform, comprising:

[0011] S1. Interact with various heterogeneous storage resources and access them through adapters that match the access protocols of each resource, and abstract the storage resources into virtual disks for unified management by the monitoring platform;

[0012] S2. Perform availability checks on the virtual disks and complete unified mounting, registering the virtual disks that pass the checks as members of the writable disk pool;

[0013] S3. When a new recording task or a recording task that needs to be restarted appears on the monitoring platform, a task scheduling cycle is triggered. During the scheduling cycle, at least two types of attributes are counted for each disk in the writable disk pool: (i) the rate of increase of used space and (ii) the current remaining available space.

[0014] S4. Determine the candidate disk set based on statistical results: Select the M disks with the slowest growth rate from the writable disk pool to form set A, and select the N disks with the largest remaining available space to form set B, thus obtaining the candidate set C = A∪B (duplicate removal); where M and N are configurable parameters.

[0015] S5. Select a target disk from the candidate set C, assign the newly added or restarted recording task to the target disk for execution and storage, and record the mapping relationship between the task and the target disk;

[0016] Thus, by unifying the abstraction of storage resources of different protocols and dynamically allocating them based on common attributes, the platform can achieve mixed use and load balancing of multiple types of storage disks and reduce the impact of single-type storage media failures on platform recording tasks.

[0017] Furthermore, the various heterogeneous storage resources in step S1 include at least two of the following three categories: storage resources based on file system protocols, including local disks and network file system NFS storage; storage resources based on object storage interfaces, including Amazon OBS (Object Storage Service) or locally built minIO small storage servers; and storage resources based on private access protocols, i.e., private storage servers provided by third-party vendors.

[0018] Furthermore, the storage resources based on the file system protocol are accessed as virtual disks in the following ways:

[0019] Local disk: First, it is formatted to adapt to the monitoring platform's file system, and then mounted to the monitoring platform's storage service via the NFS protocol;

[0020] NFS storage: Configure network parameters to establish a connection with the remote NFS storage server, configure the NFS service on the remote server and grant access permissions to the monitoring platform, and then mount it into the storage service via the NFS protocol.

[0021] Furthermore, the storage resources based on the object storage interface are abstracted into virtual disks through an object storage access interface compatible with S3. The specific process includes: first, uploading a test file to the object storage resource, then downloading the test file from the resource, comparing the integrity of the uploaded and downloaded files, and if the integrity is consistent, the abstraction and access are completed.

[0022] Furthermore, the storage resource based on the private access protocol is accessed as a virtual disk in the following way: the dedicated driver corresponding to the private storage resource is loaded on the monitoring platform, and a connection handshake and validity verification (including connection stability detection and data read / write test) are initiated according to its private protocol. After the verification is successful, the access is completed.

[0023] Furthermore, the availability detection in step S2 specifically includes: writing a test file to the heterogeneous storage resource corresponding to the virtual disk, then reading back the test file from the storage resource, and comparing the consistency of the upload and download results based on the file content digest or checksum; if the comparison is consistent, the availability detection is determined to be passed, and virtual disks that fail the detection are not added to the writable disk pool.

[0024] Furthermore, in step S3, the rate of increase in space has been defined as the rate of increase between two consecutive statistical moments. , The ratio of the difference in used disk space to the time difference is calculated using the following formula: ;in, For a moment The amount of space already used on the disk. For a moment The amount of space already used on the disk. .

[0025] Furthermore, after calculating the rate of increase in used disk space and the current remaining available space, it is necessary to perform debounce or smoothing processing on the two types of attribute data. The processing methods include sliding time window statistics and / or exponentially weighted moving averages to reduce the impact of instantaneous data fluctuations on the statistical results.

[0026] Furthermore, in step S4, M and N are configurable positive integers; when the total number of disks in the writable disk pool is less than M, set A is all disks in the writable disk pool; when the total number of disks in the writable disk pool is less than N, set B is all disks in the writable disk pool; if set A and set B have an intersection, that is, a certain disk belongs to both A and B, then the disk is deduplicated and retained once in the candidate set C.

[0027] Furthermore, in step S5, when selecting the target disk from the candidate set C, a uniform random or weighted random method is used. In weighted random, sampling is performed using a weighted combination of the candidate disk's standardized remaining available space and the inverse of its standardized growth rate, according to a preset proportional coefficient. Before triggering in S3 and selecting in S5, health thresholds, capacity thresholds, and concurrency thresholds are set as preconditions. Disks that do not meet the thresholds or exhibit connection anomalies, excessive I / O error rates, or failed availability detection are removed in the current scheduling cycle. When there are insufficient candidates, the candidate range can be expanded before selection. The scheduling cycle can be configured as a fixed duration or event-triggered.

[0028] Furthermore, the content of recording the mapping relationship between the task and the target disk in step S5 includes: the recording task number, the task start time, the estimated storage duration of the task, the target disk identifier (such as the disk unique ID and the corresponding storage resource type), and the current remaining available space of the target disk. The mapping relationship is stored in the task management database of the monitoring platform and is used for subsequent task progress tracking and task reallocation in case of disk failure.

[0029] The present invention adopts the above technical solution, and the specific beneficial effects are as follows:

[0030] In this invention, local disks, NFS cloud storage, object storage, and private protocol storage are uniformly abstracted into virtual disks and incorporated into a writable disk pool. Combined with health threshold detection and availability verification mechanisms, abnormal disks can be automatically removed within the current scheduling cycle. When a certain type of storage fails, newly added or restarted recording tasks do not need to rely on the failed medium and can still be allocated to other healthy disks to continue execution. This effectively avoids the risk of platform-wide recording interruption caused by a single type of medium failure, significantly improves the reliability of the platform storage system, and reduces the risk of single point of failure.

[0031] In this invention, the different protocols of various storage resources (such as NFS, S3, and proprietary protocols) are shielded by the adapter layer, so that the upper layer of the platform only faces a unified write interface and common attributes (capacity, health status, and usage trend). There is no need to develop and maintain independent modules for different storage media, which greatly reduces system complexity and coupling with the underlying storage vendors. Moreover, when new types of storage media are subsequently expanded, they only need to be adapted to the corresponding protocol and pass the availability test to be connected, realizing "plug and play" for multiple types of storage, with extremely strong compatibility and scalability.

[0032] In this invention, task allocation constructs a candidate set A∪B based on two indicators: "slowest rate of increase in used space (low load)" and "maximum current remaining space (sufficient capacity)". Compared with a single indicator strategy, this approach can effectively suppress the formation of hot disks and avoid large-capacity disks from being idle, thereby improving overall load balancing and capacity utilization. At the same time, sliding window statistics or exponentially weighted moving average processing is applied to the two types of attribute data to weaken the impact of instantaneous data fluctuations on scheduling decisions, reduce unnecessary oscillations and jitters in task allocation across multiple disks, and ensure scheduling stability.

[0033] In this invention, the candidate set size parameters M and N, health threshold, capacity threshold, concurrency threshold, and task scheduling cycle all support flexible configuration. They can be quickly optimized according to the differences in the number of monitoring points, code rate distribution, and storage topology. They can be adapted to various deployment scenarios from small edge nodes to large-scale central platforms without requiring significant modifications to the core logic due to changes in scenario scale, making them highly adaptable to deployment.

[0034] In this invention, two disk selection methods are supported within the candidate set: uniform random and weighted random. Uniform random can achieve basic balanced allocation of tasks, while weighted random combines the "standardized value of remaining space" and the "inverse ratio of the standardization of growth rate" into a weight according to a preset ratio. This can not only prioritize disks with large free space and slow growth, but also avoid tasks being concentrated on a single disk, thereby further improving the quality of task allocation and meeting the needs of multi-objective disk selection.

[0035] In this invention, before task allocation, a comprehensive health threshold (such as I / O error rate and connection response time), capacity threshold (such as remaining space percentage), and concurrency threshold (such as the number of tasks to be carried) are used for pre-interception to eliminate disks that do not meet the conditions. When there are not enough candidate disks or the conditions are not met, an elastic fallback path is supported to expand the candidate range (such as increasing the number of candidate disks), ensuring that a feasible task allocation scheme can still be provided when resources are scarce or there are local anomalies, thereby improving system elasticity.

[0036] In this invention, users can reuse existing storage media and support the mixed use of different types of storage media. There is no need to "force the purchase of homogeneous storage" or "bind to dedicated hardware devices" according to existing technical requirements, which greatly reduces the additional hardware investment cost. At the same time, after unified abstraction, only one set of upper-layer storage management logic needs to be maintained, and there is no need to maintain multiple sets of storage stacks in parallel, which significantly reduces the complexity and cost of operation and maintenance.

[0037] In this invention, when a new type of storage medium is connected, it is only necessary to complete the corresponding protocol adaptation and availability detection (such as file upload and download verification for object storage and protocol handshake verification for private storage) before it can be quickly included in the writable disk pool for task scheduling. It also supports on-demand rolling expansion and heterogeneous storage migration, and can complete the capacity expansion or media replacement of the storage system without downtime, reducing downtime window and migration risk, and improving system operation and maintenance flexibility.

[0038] In this invention, the mapping relationship between "recording task - target disk" is recorded in detail, including information such as task number, disk identifier, and task start time. This mapping relationship can be used for subsequent recording task progress tracking, storage anomaly location, and manual intervention. When the disk fails or needs maintenance, the platform can quickly locate the task to be reassigned based on this mapping relationship and prioritize the reassignment of the task in the next scheduling cycle, which greatly shortens the fault recovery time and enhances the maintainability of the system.

[0039] In this invention, the availability detection process adopts a verification logic of "writing test file - reading back test file - comparing integrity". By comparing file content digests or check codes, storage path configuration errors, abnormal access permissions, or media stability issues can be detected in advance. This avoids assigning recording tasks to storage media that are "connectable but unusable", effectively reducing the risk of data loss and write failure, and ensuring the consistency and security of recorded data.

[0040] In this invention, the unified abstraction layer only exposes general write interfaces and storage metrics to the upper layer, without changing the original recording task generation, management and scheduling process of the monitoring platform; the platform can directly obtain the ability to use multiple types of storage and intelligent task allocation without changing the upper layer business logic, realizing the decoupling of storage function upgrade and upper layer business, without any intrusion on the existing business. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the mixed-use method of the present invention.

[0043] Figure 2 This is a schematic diagram of a specific implementation example of the mixed-use method of the present invention. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0045] like Figure 1 As shown, this embodiment provides a method for combining unified mounting of multiple types of storage disks and dynamic task allocation in a monitoring platform, including:

[0046] S1. Interact with various heterogeneous storage resources and access them through adapters that match the access protocols of each resource, and abstract the storage resources into virtual disks for unified management by the monitoring platform;

[0047] S2. Perform availability checks on the virtual disks and complete unified mounting, registering the virtual disks that pass the checks as members of the writable disk pool;

[0048] S3. When a new recording task or a recording task that needs to be restarted appears on the monitoring platform, a task scheduling cycle is triggered. During the scheduling cycle, at least two types of attributes are counted for each disk in the writable disk pool: (i) the rate of increase of used space and (ii) the current remaining available space.

[0049] S4. Determine the candidate disk set based on statistical results: Select the M disks with the slowest growth rate from the writable disk pool to form set A, and select the N disks with the largest remaining available space to form set B, thus obtaining the candidate set C = A∪B (duplicate removal); where M and N are configurable parameters.

[0050] S5. Select a target disk from the candidate set C, assign the newly added or restarted recording task to the target disk for execution and storage, and record the mapping relationship between the task and the target disk;

[0051] Thus, by unifying the abstraction of storage resources of different protocols and dynamically allocating them based on common attributes, the platform can achieve mixed use and load balancing of multiple types of storage disks and reduce the impact of single-type storage media failures on platform recording tasks.

[0052] As one implementation method, the various heterogeneous storage resources in step S1 of this embodiment include at least two of the following three categories: storage resources based on file system protocols, including local disks and Network File System (NFS) storage; storage resources based on object storage interfaces, including Amazon OBS (Object Storage Service) or locally built minIO small storage servers; and storage resources based on private access protocols, i.e., private storage servers provided by third-party vendors. This classification covers common storage scenarios in monitoring platforms, avoids compatibility limitations caused by omissions in storage types, and provides users with a basis for "on-demand selection," which aligns with the core objective of this invention: "expanding the range of storage choices."

[0053] As one implementation method, the storage resources based on the file system protocol described in this embodiment are accessed as virtual disks in the following way:

[0054] Local disk: First, it is formatted to adapt to the monitoring platform's file system, and then mounted to the monitoring platform's storage service via the NFS protocol. Formatting allows the local disk to adapt to the monitoring platform's file system (such as EXT4 or NTFS), avoiding mounting failures or read / write anomalies caused by file system incompatibility, and laying the foundation for subsequent NFS protocol mounting.

[0055] NFS Storage: Configure network parameters to establish a connection with the remote NFS storage server. Configure the NFS service on the remote server and grant access permissions to the monitoring platform. Then, mount the data to the storage service via the NFS protocol. Configuring the NFS service and granting access permissions ensures a secure and stable connection between the monitoring platform and the NFS cloud storage server, avoiding storage access failures due to permission issues. Simultaneously, network parameter configuration (such as IP address and port) guarantees low latency for remote connections, meeting the "real-time storage" requirement of recording tasks.

[0056] As one implementation method, the storage resource based on the object storage interface described in this embodiment is abstracted into a virtual disk through an object storage access interface compatible with S3. The specific process includes: first, uploading a test file to the object storage resource, then downloading the test file from the resource, and comparing the integrity of the uploaded and downloaded files. If the integrity is consistent, the abstraction access is completed. Verifying file integrity through MD5 digest comparison (or CRC32 checksum) can detect network packet loss, abnormal permissions, and other issues during the object storage access process in advance, ensuring that the accessed object storage is "readable and writable normally," avoiding data loss in subsequent recording tasks due to storage access problems, and providing reliable protection for object storage to participate in recording task allocation.

[0057] As one implementation method, the storage resource based on the private access protocol described in this embodiment is accessed as a virtual disk in the following way: The monitoring platform loads the dedicated driver corresponding to the private storage resource, initiates a connection handshake and validity verification (including connection stability detection and data read / write tests) according to its private protocol, and completes the access after successful verification. Loading the dedicated driver is crucial for adapting to the private protocol, ensuring that the monitoring platform can correctly parse the private protocol instructions and avoid access failures due to protocol incompatibility. Connection stability detection and data read / write tests simulate the storage scenario of actual recording tasks, verify the reliability of the private storage path, and avoid recording interruptions due to storage instability, which aligns with the present invention's objective of "improving platform reliability."

[0058] As one implementation method, the availability detection in step S2 of this embodiment specifically includes: writing a test file to the heterogeneous storage resource corresponding to the virtual disk, and then reading back the test file from the storage resource; comparing the consistency of the upload and download results based on the file content digest or checksum; if the comparison is consistent, the availability detection is determined to be passed, and virtual disks that fail the detection are not added to the writable disk pool. This detection logic can simulate the entire "write-read" process of an actual recording task, pre-screening storage media that are "connectable but unusable" (such as disks with incorrect path configuration or hardware failure), avoiding the allocation of recording tasks to invalid storage, and reducing the risk of data loss; at the same time, disks that fail the detection are not added to the writable disk pool, which can reduce invalid judgments in subsequent task scheduling and improve scheduling efficiency.

[0059] As one implementation method, in this embodiment, the rate of increase of the space used in step S3 is defined as the rate of increase between two consecutive statistical moments. , The ratio of the difference in used disk space to the time difference is calculated using the following formula: ;in, For a moment The amount of space already used on the disk. For a moment The amount of space already used on the disk. This formula quantifies disk load trends using the "difference / time difference," which better reflects the actual load growth of the disk compared to "only counting used space." For example, if two disks both have 50GB of free space, the disk with a load increase rate of 1GB / minute has a higher load. This formula can accurately distinguish such differences, providing a quantitative basis for subsequent "low-load disk screening."

[0060] As one implementation method, after statistically analyzing the rate of increase in used disk space and the current remaining available space, this embodiment requires performing debouncing or smoothing processing on the two types of attribute data. The processing methods include sliding time window statistics and / or exponentially weighted moving averages to reduce the impact of instantaneous data fluctuations on the statistical results. Smoothing processing can weaken the interference of instantaneous fluctuations (such as a sudden surge in the rate of increase caused by a large file write at a certain moment) on the statistical results. For example, if a disk experiences a rate of increase of 5GB / minute within one minute due to instantaneous writes, sliding time window statistics (taking a 5-minute average) can bring it back to the true load (such as 1GB / minute), ensuring that the statistical data reflects the long-term state of the disk, avoiding the problem of "misselecting high-load disks" due to data distortion, and improving scheduling accuracy.

[0061] As one implementation method, in step S4 of this embodiment, M and N are configurable positive integers; when the total number of disks in the writable disk pool is less than M, set A is all disks in the writable disk pool; when the total number of disks in the writable disk pool is less than N, set B is all disks in the writable disk pool; if set A and set B have an intersection, that is, a disk belongs to both A and B, then the disk is deduplicated and retained once in the candidate set C. M and N are configurable (e.g., the default M=N=3), which can adapt to monitoring platforms of different sizes—small platforms (less than 10 disks) can set M=N=1 to avoid the candidate set being too large; large platforms (more than 50 disks) can set M=N=5 to ensure that the candidate set covers a sufficient number of low-load, high-capacity disks; deduplication processing can prevent the same disk from repeatedly entering the candidate set, reduce redundant judgments when selecting disks, and at the same time ensure the uniqueness of the candidate set C, improving task allocation efficiency.

[0062] In one implementation, step S5 in this embodiment selects the target disk from the candidate set C using a uniform random or weighted random method. In weighted random sampling, the sampling weights are based on a preset ratio coefficient combining the standardized remaining available space of the candidate disk with the inverse ratio of the standardized increase rate. Before triggering S3 and selecting in S5, health thresholds, capacity thresholds, and concurrency thresholds are set as preconditions. Disks that do not meet the thresholds or exhibit connection anomalies, excessive I / O error rates, or failed availability detection are removed during the current scheduling cycle. When there are insufficient candidates, the candidate range can be expanded before selection. The scheduling cycle can be configured as a fixed duration or event-triggered. Uniform random is suitable for small platforms with general load balancing requirements (such as cameras with 20 or fewer channels), which can simplify disk selection logic and reduce the consumption of platform computing resources; weighted random uses weight combinations (such as "remaining space × 0.6 + inverse growth rate × 0.4") to balance "low load" and "high storage capacity", which can prioritize the selection of better disks and improve storage utilization; pre-threshold interception can exclude unhealthy disks from the source and avoid tasks being assigned to faulty storage; expanding the range when there are insufficient candidates can ensure that there are still feasible storage allocation solutions in extreme scenarios (such as the failure of most disks) and improve platform elasticity.

[0063] As one implementation method, the content of recording the mapping relationship between the task and the target disk in step S5 of this embodiment includes: the recording task number, the task start time, the estimated storage duration of the task, the target disk identifier (such as the disk's unique ID and the corresponding storage resource type), and the current remaining available space of the target disk. The mapping relationship is stored in the task management database of the monitoring platform for subsequent task progress tracking and task reallocation in case of disk failure. Recording the mapping relationship enables full lifecycle traceability of "task-disk"—for example, when a camera recording task is interrupted, the disk where it is stored can be quickly located by the task number to investigate the cause of the failure; when the disk fails, unfinished tasks on the disk can be identified in batches based on the mapping relationship and prioritized for reallocation in the next scheduling cycle, shortening the failure recovery time; at the same time, recording the remaining available space of the target disk can help determine whether the disk can bear new tasks during subsequent scheduling, avoiding "disk overload".

[0064] like Figure 2As shown, in this embodiment, access processing is first performed on four types of heterogeneous storage resources: local disks, NFS cloud storage, object storage, and private storage. For local disks, the operations of installing the disk on the server, formatting the disk, and configuring the NFS service are performed sequentially. For NFS cloud storage, an NFS cloud storage server is purchased or configured first, then the network is configured and a connection to the remote server is established. For object storage (such as Amazon OBS, locally built minIO servers, etc.), after the purchase or configuration of the server and the configuration of the network connection are completed, access verification is completed by uploading test files, downloading test files, and comparing the integrity of the test files. For private storage, after the purchase or configuration of a private protocol storage server and the configuration of the network connection are completed, a dedicated driver is loaded and connection verification is completed through the private protocol. After all types of storage resources are accessed, the monitoring platform uniformly performs the storage disk mounting operation. When a new recording task or a recording task that needs to be restarted is added to the monitoring platform, the task allocation process is triggered: on the one hand, the disk with the slowest rate of increase in used space in the writable disk pool is counted (if the number of disks exceeds a preset threshold, the top few disks are selected to form set A); on the other hand, the disk with the largest remaining available space is counted (if the number of disks exceeds a preset threshold, the top few disks are selected to form set B); finally, a disk is randomly selected from set A and set B, and the new or restarted recording task is allocated to that disk for storage.

[0065] This embodiment aims to solve the problem of "requiring consistent storage media" in the recording storage of existing video surveillance platforms: existing technologies either force users to use the same type of storage media or bind specific hardware storage media, which causes hardware or network failures of a single storage media to affect all recordings on the entire platform, and users need to purchase additional specified storage media, resulting in a narrow selection range and high costs.

[0066] like Figure 2As shown, various storage media, including local disks, ordinary cloud storage, object storage service type storage, and private storage servers from other vendors, are first abstracted and treated as disks for mounting operations. For local disks directly installed on the monitoring platform server, the disk is first formatted and then mounted to the monitoring platform's storage service (OSS) via the NFS (Network File System) protocol. For ordinary cloud storage, the mounting logic is similar to that of local disks: a network connection is first established with the corresponding remote server, and then the cloud storage is mounted to the monitoring platform's storage service via the NFS protocol. For object storage service type storage (such as Amazon OBS (Object Storage Service) and locally built minIO small storage servers), access is initiated using the S3 protocol: a test file is uploaded to the storage, and after the upload is complete, the test file is downloaded. The consistency between the uploaded and downloaded files is compared; if the comparison results are consistent, the storage is considered mounted normally. For various private storage servers provided by other vendors, the connection and mounting are completed using the appropriate access method according to the server's corresponding private access protocol.

[0067] After all types of storage media are successfully mounted, they become disks that the monitoring platform can write to. If the monitoring platform has multiple of these disks mounted, the platform will allocate recording storage tasks based on the disks' general attributes: During the interval between two task scheduling, the usage increase rate of each mounted disk is calculated, and the disk with the slowest usage increase rate is marked (denoted as disk A). If the number of mounted disks exceeds a preset number, the disks with the slowest usage increase rate are marked and together form disk group A. At the same time, the current remaining space of each mounted disk is calculated, and the disk with the largest current remaining space is marked (denoted as disk B). If the number of mounted disks exceeds a preset number, the disks with the largest current remaining space are marked and together form disk group B. Finally, a disk is randomly selected from disk A (or disk group A) and disk B (or disk group B), and the newly added recording task is assigned to that disk for storage.

[0068] Through the above process, when a certain type of storage medium fails, it will not affect the recording tasks of the monitoring platform on other types of storage media, effectively improving the overall reliability of the platform. At the same time, users can flexibly choose storage media according to their actual needs (choose ordinary NFS disks when the recording storage scale is small, choose S3 protocol storage when pursuing stable storage performance, and choose private storage protocol disks when there are personalized needs), which significantly expands the range of storage media choices for platform users.

[0069] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for monitoring the mixed use of uniform mounting and dynamic task allocation of multiple types of storage disks in a platform, characterized in that: The method comprises the following steps: ​ S1, access a plurality of heterogeneous storage resources through adapters matched with respective resource access protocols, and abstract the storage resources into virtual disks for unified management by a monitoring platform; S2, perform availability detection on the virtual disks and complete unified mounting, and register the virtual disks detected through the detection as members in a writable disk pool; S3, trigger a task scheduling period when a new recording task appears on the monitoring platform or a recording task needs to be restarted, and in the scheduling period, count at least two types of attributes for each disk in the writable disk pool: (i) the rising speed of the used space, and (ii) the current remaining available space; S4, determine a candidate disk set based on the counting results: select the first M disks with the slowest rising speed from the writable disk pool to form set A, and select the first N disks with the largest remaining available space to form set B, and obtain the candidate set C = A ∪ B; wherein M and N are configurable parameters; S5, select a target disk from the candidate set C, and assign the new or restarted recording task to the target disk for storage, and record the mapping relationship between the task and the target disk. Thus, through unified abstraction of different protocol storage resources and dynamic allocation based on general attributes, mixed use and load balancing of multiple types of storage disks are realized, and the influence of failure of a single type of storage medium on platform recording tasks is reduced.

2. The method of claim 1, wherein: The plurality of heterogeneous storage resources in step S1 at least include at least two of the following three types: storage resources based on a file system protocol; storage resources based on an object storage interface; storage resources based on a private access protocol.

3. The method of claim 2, wherein: The storage resources based on the file system protocol are accessed as virtual disks through the following ways: Local disk: first, format it to adapt to the file system of the monitoring platform, and then mount it to the inside of the storage service of the monitoring platform through the NFS protocol; NFS storage: configure network parameters to establish a connection with the remote server of the NFS storage, configure the NFS service on the remote server and open the access permission of the monitoring platform, and then mount it to the inside of the storage service through the NFS protocol.

4. The method of claim 2, wherein: The storage resources based on the object storage interface are abstracted as virtual disks through an object storage access interface compatible with S3, and the specific process includes: first, upload a test file to the object storage resource, then download the test file from the resource, and compare the integrity of the uploaded and downloaded files. If the integrity is consistent, the abstraction access is completed.

5. The method of claim 2, wherein: The storage resources based on the private access protocol are accessed as virtual disks through the following ways: load the exclusive driver corresponding to the private storage resource on the monitoring platform, initiate connection handshake and validity verification according to the private protocol, the validity verification includes connection stability detection and data read-write test, and the access is completed after the verification is passed.

6. The method of claim 1, wherein: The availability detection in step S2 specifically includes: writing a test file to the heterogeneous storage resource corresponding to the virtual disk, and reading the test file from the storage resource, and comparing the consistency of the uploaded and downloaded results based on the file content digest or check code; if the comparison is consistent, it is determined that the availability detection is passed, and the virtual disks that do not pass the detection are not added to the writable disk pool.

7. The method of claim 1, wherein: The rising speed of the used space in the step S3 is defined as the ratio of the used space difference of the disk and the time difference at the two continuous statistics time points , , and the calculation formula is: ; wherein, is the used space of the disk at the time point , is the used space of the disk at the time point , .

8. The method of claim 1 or 7, wherein: After the rising speed of the used space of the statistical disk and the current remaining available space are calculated, the two types of attribute data are subjected to de-bouncing or smoothing processing, such as sliding time window statistics and / or exponential weighted moving average, to reduce the influence of instantaneous data fluctuation on the statistical result.

9. The method of claim 1, wherein: In the step S4, M and N are configurable positive integers; when the total number of disks in the writable disk pool is less than M, the set A is all the disks in the writable disk pool; when the total number of disks in the writable disk pool is less than N, the set B is all the disks in the writable disk pool; if the set A and the set B have an intersection, that is, a disk belongs to both A and B, the disk is retained once in the candidate set C.

10. The method according to any one of claims 1 to 7 or claim 9, characterized in that: In the step S5, a uniform random or weighted random method is used to select the target disk from the candidate set C; in the weighted random method, the inverse ratio of the normalized remaining available space and the normalized rising speed of the candidate disk is combined with a preset proportion coefficient to obtain a weight for sampling; and before the step S3 is triggered and the step S5 is selected, a health threshold, a capacity threshold and a concurrency threshold are set as precondition, and the disks that do not meet the threshold or have connection abnormity, I / O error rate exceeding limit, availability detection failure are excluded in the current scheduling period; when the candidate is insufficient, the candidate range is expanded and then the selection is performed. The scheduling period can be configured as a fixed time length or an event trigger.

Images