A method, system, device, and storage medium for generating and storing metadata

Abstract

The application provides a method, system, device and storage medium for generating and storing metadata, the method comprising: defining multiple types of metadata and setting an association relationship between different types of metadata; generating different types of metadata according to the definition and managing metadata of a corresponding type through different service processes; and storing different types of metadata into different metadata databases and adopting different redundancy protection strategies for different metadata databases to ensure the consistency of metadata. The application associates different types of video service metadata in a one-to-many manner, separates the storage system into different service processes to generate and manage different types of metadata, and different service processes undertake different service functions.

Description

Technical Field

[0001] This invention relates to the field of security monitoring technology, and more specifically, to a method, system, device, and storage medium for generating and storing metadata. Background Technology

[0002] Existing technologies employ separate storage of metadata and data to achieve an efficient storage service layer. However, regardless of the implementation or optimization scheme, each storage node service must generate and manage all types of video service metadata, leading to long recording retrieval times. Both schemes require multiple I / O operations to locate the required reading position. Furthermore, neither approach can provide a high-performance, highly reliable storage system with low resource consumption in terms of metadata reliability. Current implementations all suffer from the following problems:

[0003] 1. The way metadata is generated and managed means that video retrieval requires searching all storage nodes and merging the results from all storage nodes before returning them to the caller, making it impossible to achieve second-level return for video retrieval. Reading video data requires multiple I / O operations to locate the reading position, making it impossible to read the video in a maximum of two I / O operations.

[0004] 2. There are also limitations in business failover and space load balancing. For example, if a storage node fails, the storage services on the failed node will be switched to other storage nodes. After the failed node recovers, the storage services in the system will be load balanced, and some storage services on other storage nodes will be switched to the newly recovered node. This situation will result in some channels occupying unfilled storage space during the storage cycle not being released, resulting in wasted space.

[0005] 3. A dedicated metadata server or a server node with high configuration is required to store metadata in order to provide a high-performance, highly reliable distributed storage system. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a method, system, electronic device, and computer-readable storage medium for generating and storing metadata. This invention associates different types of video service metadata in a one-to-many manner and separates different service processes in the storage system to generate and manage different types of metadata. Different service processes undertake different business functions. While ensuring high performance in data reading and writing, the reliability, scalability, and compatibility of the video storage system are effectively improved and enhanced using lower resources.

[0007] To achieve the above objectives, one aspect of the present invention provides a method for generating and storing metadata, comprising the following steps: defining multiple types of metadata and setting association relationships between different types of metadata; generating different types of metadata according to the definitions and managing the corresponding types of metadata through different service processes; and storing different types of metadata in different metadata databases and employing different redundancy protection strategies for different metadata databases to ensure the consistency of metadata.

[0008] In some implementations, the various types of metadata defined include: dividing metadata into node information, channel information, block file information, and time period information. The node information is used to describe node service information, the channel information is used to describe camera recording information, the block file information is used to describe the storage resource information allocated by the node service, and the time period information is used to describe the attributes, type, and location of the video data.

[0009] In some implementations, setting the association between different types of metadata includes: setting one channel information to correspond to multiple block file information, and setting one block file information to correspond to multiple time period information.

[0010] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: setting node information and channel information as global metadata, and setting block file information and time period information as local node metadata; and managing the global metadata through a central service and managing the local node metadata through a node service.

[0011] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: periodically detecting all channel information metadata through a central service, and detecting the time period list of each channel information metadata to determine whether the time period status is locked; in response to the time period status not being locked and the central service detecting that the end time of a time period exceeds the channel's storage period, sending an instruction to the node service where the time period is located to evict the time period through the central service; in response to the node service receiving the instruction, updating the status of the time period information metadata according to the time period identifier; and clearing the time period information from the channel information metadata and updating the channel information metadata to the database.

[0012] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: periodically detecting all block file information metadata and time period information metadata through the node service to determine whether the current block file is currently undergoing read or write operations; in response to the current block file not undergoing read or write operations, detecting the status of all time period information metadata on the block file through the node service and determining whether the status of all time period information metadata is in an obsolete state; in response to the time period information metadata being in an obsolete state, reclaiming the block file as an idle block through the node service and deleting all time period information metadata on the block file; and updating the reclaimed block file information metadata through the node service and simultaneously synchronizing the metadata information with the central service.

[0013] In some implementations, storing different types of metadata in different metadata databases and employing different redundancy protection strategies for different metadata databases to ensure metadata consistency includes: using a clustered database to store the global metadata and using a dual-backup database to store the local node metadata.

[0014] In another aspect of this invention, a system for generating and storing metadata is provided, comprising: a definition module for defining multiple types of metadata and setting association relationships between different types of metadata; a management module for generating different types of metadata according to the definitions and managing the corresponding types of metadata through different service processes; and a storage module for storing different types of metadata in different metadata databases and employing different redundancy protection strategies for different metadata databases to ensure the consistency of metadata.

[0015] In another aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing the steps of the method described above.

[0016] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0017] The present invention has the following beneficial technical effects: it associates different types of video service metadata in a one-to-many manner, and separates different service processes in the storage system to generate and manage different types of metadata. Different service processes undertake different business functions. While ensuring high performance in data reading and writing, it effectively improves and enhances the reliability, scalability, and compatibility of the video storage system with lower resources. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating an embodiment of the method for generating and storing metadata provided by the present invention;

[0020] Figure 2 The video storage service system structure diagram provided by this invention;

[0021] Figure 3 A diagram showing the relationship between channels, block files, and time periods provided by this invention;

[0022] Figure 4 This is a schematic diagram illustrating the relationship between the central service and the central metadata database provided by the present invention.

[0023] Figure 5 This is a schematic diagram illustrating data backup at a node provided by the present invention;

[0024] Figure 6 This is a schematic diagram illustrating data backup in a database as provided by the present invention;

[0025] Figure 7 A flowchart for data writing provided by this invention;

[0026] Figure 8 A flowchart for data reading provided by the present invention;

[0027] Figure 9 A flowchart for video recording locking / unlocking provided by the present invention;

[0028] Figure 10 A flowchart for data elimination provided by this invention;

[0029] Figure 11 A flowchart for metadata reconstruction provided by this invention;

[0030] Figure 12 A schematic diagram illustrating an embodiment of the system for generating and storing metadata provided by the present invention;

[0031] Figure 13 A schematic diagram of the hardware structure of an embodiment of an electronic device for generating and storing metadata provided by the present invention;

[0032] Figure 14A schematic diagram illustrating an embodiment of a computer storage medium for generating and storing metadata provided by the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0034] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0035] A first aspect of the present invention provides an embodiment of a method for generating and storing metadata. Figure 1 The diagram shown is an embodiment of the method for generating and storing metadata provided by the present invention. Figure 1 As shown, the embodiments of the present invention include the following steps:

[0036] S1. Define various types of metadata and set the association relationships between different types of metadata;

[0037] S2. Generate different types of metadata according to the definition, and manage the corresponding types of metadata through different service processes; and

[0038] S3. Store different types of metadata in different metadata databases and use different redundancy protection strategies for different metadata databases to ensure metadata consistency.

[0039] Real-time video storage in the security monitoring industry involves recording real-time video streams from cameras 24 / 7 for subsequent applications such as playback, download, and computation. Based on these industry characteristics, video storage systems are generally divided into a storage service layer and a storage data layer. The storage service layer is responsible for the storage management of video stream data, undertaking video storage business functions, generating and managing metadata related to video services, interfacing with the storage data layer, allocating resources provided by the storage data layer, and recording relevant data information for efficient data retrieval later. The storage data layer provides physical storage space for data, ensuring even distribution of data, providing security protection for stored data, and guaranteeing I / O performance for data read / write. The storage data layer typically includes storage resource pools provided by physical disks, file systems, block devices, object storage, and third-party storage systems. Because the storage service layer must support video service functions, ensure the characteristics of a distributed storage system, and effectively integrate with the storage data layer, the generation and management of metadata plays a crucial role in the performance, reliability, availability, scalability, and compatibility of the storage system.

[0040] This invention categorizes metadata into multiple types and associates different types of video service metadata using a one-to-many approach. Different service processes are separated to generate and manage these different types of metadata, each performing a different function. This avoids the situation where each storage node service generates and manages all types of video service metadata, handles all business processing, and responds to client requests, leading to performance degradation. For example, when retrieving data, all storage nodes must be searched, and the results from all storage nodes must be merged before being returned to the caller, making it impossible to achieve second-level data retrieval. The metadata in this invention is stored in a database, with different redundancy protection strategies employed by different metadata databases to ensure metadata consistency. The metadata is distributed across all nodes in the system, eliminating the need for a separate metadata server and reducing additional server costs. Furthermore, this invention optimizes and improves metadata reliability and data read / write performance.

[0041] Figure 2 The video storage service system structure diagram provided by this invention is as follows: Figure 2 As shown, the video storage service system comprises five service modules: SDK, central service, central metadata database, node service, and local node metadata database. Each storage node server runs these service modules, and each individual storage node server is a relatively independent storage system, supporting all the functional features of the system and capable of handling any operation of the SDK. Combined with... Figure 2 The embodiments of the present invention will be described.

[0042] The SDK provides APIs (Application Programming Interfaces) for the management platform, simplifies and encapsulates the functions used by the management platform, and ultimately provides static / dynamic link libraries for the management platform to use.

[0043] The central service possesses key functions such as high availability, cluster management, load balancing, channel management, and business processing. It manages all node services within the system and performs unified, balanced allocation of storage resources provided by the storage data layer. It supports multiple management platforms using the system, avoiding redundant channel creation by management platforms. It only handles command operations, not data read / write operations. The central service cluster elects a master service through an election algorithm to perform its assigned functions. This master service generates and manages global metadata for node and channel information within the system, ensuring data consistency.

[0044] The central metadata database uses a database cluster to store global metadata managed by the central service. Only one main central service operates the database, and the database cluster ensures data consistency and high availability within the database.

[0045] The node service is responsible for synchronizing current node information with the central service, managing time periods, managing block files, and performing data read / write operations. The management platform directly reads and writes data to the node service without intermediary services, thus avoiding performance bottlenecks in data read / write. The node services on each node server in the system are independent of each other, with no master-slave distinction. Each node service only manages its own allocated storage resources and manages only its own block file information and time period information metadata. Because the block file information and time period information metadata need to be updated frequently during data writing, to avoid all node services concurrently updating metadata in the database, each node service stores its own block file information and time period information metadata in a local node metadata database running on its current node server.

[0046] The local node metadata database is a single-point database used to store metadata about block files and time periods generated and managed by the node service. All local node metadata databases running on the system's node servers are independent of each other; they do not form a cluster and only communicate and interact with the node service running on the current node server.

[0047] Define various types of metadata and set the relationships between different types of metadata.

[0048] In some implementations, the various types of metadata defined include: dividing metadata into node information, channel information, block file information, and time period information. The node information is used to describe node service information, the channel information is used to describe camera recording information, the block file information is used to describe the storage resource information allocated by the node service, and the time period information is used to describe the attributes, type, and location of the video data.

[0049] The video storage service system of this invention categorizes metadata into four types: node information, channel information, block file information, and time period information. The metadata structure is defined as a key-value pattern. Node information primarily describes the size of storage resources allocated to node services and information related to node service status, used for node service status monitoring and storage resource load balancing. Channel information, block file information, and time period information are video service-related metadata, and they are interconnected. The smallest unit of metadata is only the time period; there is no metadata for frame data packets or frame data, nor is there an index file for frame data; only block files storing frame data exist. This approach significantly improves data write performance. To avoid data retrieval performance degradation due to multiple jumps in data block file location, the time period information records a list of key frame information contained within that time period.

[0050] In some implementations, setting the association between different types of metadata includes: setting one channel information to correspond to multiple block file information, and setting one block file information to correspond to multiple time period information. Figure 3 The relationship diagram of channels, block files, and time periods provided by this invention is as follows: Figure 3 As shown, there is a one-to-many relationship between channel information and block file information, and also a one-to-many relationship between block file information and time period information. Each channel can use multiple block files, and each block file belongs to only one channel. Once a block file is occupied by a channel, it cannot be used by other channels. A block file contains one or more time periods, and a time period can only exist in one block file and cannot exist across block files.

[0051] The metadata definition details are as follows:

[0052] (1) Node Information

[0053] Key: Node name (HostName).

[0054] Value: Node address, node status (online, offline), total number of block files, number of free block files.

[0055] (2) Channel Information

[0056] Key: Channel Identifier (ChannelName), which is entered by the user and guaranteed to be unique. It is usually the channel ID or device ID.

[0057] Value: Loop type, loop period, list of occupied block files (block ID, node address), list of unfilled block files (block ID, node address), list of time periods [time period ID, start time, end time, recording type identifier, node address, time period status (normal, expired, locked)].

[0058] (3) Block file information

[0059] Key: Block Identifier (BlockId), a string concatenated from the node name and the file name (which can be an incrementing number).

[0060] Value: Node address, block file name, file alignment size, file size, file engine, channel identifier (ChannelName), number of time periods, first frame time, last frame time, last frame offset address, block file status (free block, full, not full, evicted).

[0061] (4) Time period information

[0062] Key: SegmentId, a string concatenated from the block file identifier and the time segment name (which can be an incrementing number).

[0063] Value: Node address, start time, end time, channel identifier (ChannelName), block file identifier (BlockId), time period status (normal, eviction, locked), keyframe list, frame time, and offset address in the block file.

[0064] Different types of metadata are generated according to the definition, and the corresponding types of metadata are managed through different service processes.

[0065] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: setting node information and channel information as global metadata, and setting block file information and time period information as local node metadata; and managing the global metadata through a central service and managing the local node metadata through a node service.

[0066] Node information and channel information are both types of metadata that are global metadata, managed by a central service. This central service operates in a clustered mode and is highly available. Figure 4 This is a schematic diagram illustrating the relationship between the central service and the central metadata database provided by the present invention, as follows: Figure 4As shown, each server node in the cluster runs the central service process. However, all central service nodes in the cluster will elect a master central service through a master election algorithm to ensure that only one master central service handles business and manages the metadata of node information and channel information during system operation. This requires the central service metadata database to be in cluster mode, and the database itself internally ensures the consistency and reliability of node information and channel information metadata (data protection).

[0067] The two types of metadata, block file information and time period information, are local node metadata and are managed by the node service. The node service is in standalone mode. The node service running on each server node in the cluster is independent of each other. They each manage their own node's block file and time period metadata independently and without any connection. Therefore, the database does not need to be in cluster mode. Figure 5 This is a schematic diagram illustrating data backup at a node provided by the present invention, such as... Figure 5 As shown, in order to ensure the reliability of metadata, the metadata of each node also needs to be backed up on another node. The backup node can be the next node after the node IP is sorted in ascending order. The node service metadata of the first node is backed up by the second node, the node service metadata of the second node is backed up by the third node, and so on. The node service metadata of the last node is backed up by the first node.

[0068] Figure 6 This is a schematic diagram illustrating data backup in a database provided by the present invention, such as... Figure 6 As shown, data backup is also handled internally by the database itself.

[0069] Node information metadata primarily describes information related to node services in the video storage service system, including node service addresses, the amount of storage resources allocated to node services, and node service status. This node information metadata is stored in a central metadata database cluster and is centrally managed by the main central service. During runtime, the main central service loads the node information metadata from the central database into memory and periodically persists and updates the node information in memory to the central metadata database cluster. The node information metadata is mainly used for node service status monitoring and storage resource load balancing.

[0070] The main central service manages all node services in the cluster, monitoring their status and resource information. After starting, each node service registers with the central service, synchronizing its node address, total number of block files, and number of free block files. Upon receiving the registration and synchronized node information, the central service sets the node's status to online and updates its metadata. During subsequent operation, node services periodically send keep-alive heartbeats to the central service. If the central service does not receive a keep-alive heartbeat from a node within the specified time interval, it sets that node's status to offline.

[0071] The specific steps are as follows:

[0072] (1) After the node service starts, it will register with the main center service and report the total number of block files it manages and the number of free block files to the center service. The main center service will use the node name of the node service as the KEY of the node information metadata to determine whether the node information metadata exists. If it does not exist, the node information metadata of the node service will be generated. The node name will be used as the KEY, and the VALUE will be assigned the node service address, the total number of block files, the number of free block files, and the online status. The metadata information will then be stored in the central metadata database.

[0073] (2) During the operation of the node service, it will keep heartbeat with the central service. When the total number of block files managed by the node service and the number of free block files change, it will report to the main central service in real time to update the node information metadata of the node service. When the main central service determines that the VALUE value of the node information metadata of the node service has changed, it will update the node information metadata of the node service stored in the central metadata database.

[0074] (3) If the main service does not receive heartbeat keep-alive information from a node service within a certain time interval, it will consider the node service offline, locate the node information metadata of the node service, set the status value of the node information metadata to offline, and report an offline alarm for the node service. During this period, the main service will not load any business onto the node service. When the main service receives the registration or keep-alive information of the node service again, it will set the status of the node service to online, update the node information metadata in the central metadata database, and subsequently periodically receive information reported by the node service. When it determines that the VALUE value of the node information metadata of the node service has changed, it will update the node information metadata of the node service stored in the central metadata database. When there is a business request, the business will be loaded onto the node service according to the load balancing strategy.

[0075] Ideally, cluster performance is positively linearly correlated with the sum of the performance of each node. That is, the I / O and capacity of a storage cluster consisting of 10 nodes are the sum of the I / O and capacity of each of the 10 nodes. However, because load balancing cannot achieve absolute equilibrium, space utilization decreases as the number of nodes increases, and there are also external environmental factors such as network connectivity, making it difficult for overall performance to achieve linear growth. The video storage service system of this invention uses a main center service to generate and manage node information metadata, performs unified load balancing of storage resources for all node services in the system, and ensures the consistency and reliability of node information metadata, thus achieving relative balance to the greatest extent possible.

[0076] The load balancing of the video storage service system of this invention is based on the following points: 1. The number of free blocks on a node is used as the basis for judging the load size. 2. SDK channel write requests are preferentially allocated to node services running on the same server node for data writing, minimizing network transmission of data between server nodes. 3. Each channel should ideally occupy only one incomplete block file at present. If a channel occupies one or more incomplete block files due to node failure, the incomplete block files should be filled first when the channel requests to write to the node to avoid wasting space, because once a block file is occupied by a channel, other channels will no longer be able to use it.

[0077] A channel identifies a camera that generates a real-time video stream. Channel information metadata primarily describes the recording information of a specific camera within the video storage system. The channel name serves as the key for the channel information metadata, and it is also a unique identifier for each channel. In this invention's video storage system, multiple channels with the same name are not supported; therefore, the channel name is created and passed in by the user, who is responsible for ensuring its uniqueness. The channel information metadata's value includes basic information such as the loop type and storage period, as well as information about the storage space resources occupied by the channel's recordings and the recording time range. The channel information metadata is stored in a central metadata cluster and is uniformly operated by the main central service. During runtime, the main central service loads the channel information metadata from the central database into memory and periodically updates the channel information in memory to the central metadata cluster. The channel information metadata is mainly used for other video storage business-related processing besides data reading and writing, such as recording retrieval, recording locking / unlocking, and recording information eviction.

[0078] The video storage service system of this invention only records existing channels. That is, a channel must be created in the system before data can be written; otherwise, data writing will be impossible. Users create a channel by inputting the channel name, loop type, and storage period into the SDK and submitting it to the main central service. The main central service uses the channel name as the key and assigns values ​​to the loop type and storage period fields in the VALUE to generate the channel's metadata. If the channel's metadata already exists, the central service will return an error message to the SDK indicating that the channel already exists, ensuring that channels in the cluster are not created repeatedly. In this case, all fields in the channel's metadata except for the loop type and storage period are empty.

[0079] When a user writes data to the storage system via the SDK, the SDK first requests a write node for that channel from the main service center. After the main service center allocates a write node for the channel based on the load balancing strategy related to the node information metadata, the SDK directly interacts with the node service on the allocated write node to write data. When the node service allocates a free block file for the channel, it synchronizes the block file information occupied by the channel with the main service center, which then updates the field values ​​related to the occupied block file in the channel information metadata. When data writing begins, the node service generates a time period and synchronizes it with the main service center. The main service center updates the field values ​​related to the time period in the channel information metadata. Subsequent data writing continues without further interaction with the main service center. When the time period ends, the node service synchronizes the information for that time period with the main service center again. If the main service updates the time-related field values ​​of the channel information metadata again, the SDK and node service will not interact with the main service during continuous data writing if the current block file is not full, in order to ensure the performance of data writing. When the block file is full, the node service synchronizes the block file information currently occupied by the channel with the main service. The main service will update the block file occupied field values ​​of the channel information metadata, and the SDK will request a write node for the channel from the main service again. The main service will allocate a write node for the channel again according to the load balancing strategy related to the node information metadata. The SDK will then directly interact with the node service on the allocated write node to write data. The node service will synchronize information with the main service in the same way as above, and the main service will update the channel information metadata of the channel.

[0080] When a node service fails, the information currently being written to the channel cannot be synchronized to the main center service, leading to data consistency issues in the channel information metadata. There are two strategies to resolve this data consistency issue. One is that when the main center service responds to user business commands, if it finds that some metadata field values ​​of the channel are inconsistent, it will obtain the latest information from the node service where the information is located. Even if the node service fails, it can obtain information from the backup service of the failed node service. The other is that after the failed node service recovers, it will synchronize the unsynchronized information to the main center service to resolve the data consistency issue of the channel information metadata.

[0081] When the main service receives a recording request for a specific channel, it queries the time period list in the channel's metadata based on the start time, end time, and recording identifier. It then compares the information in the time period list with the search criteria and returns a list of time periods that meet the search conditions to the SDK. The SDK then directly communicates with the node service on the server node containing the time period to read the data, using the time period ID, node address, start time, and end time from the time period list. Because recording retrieval only requires the main service to compare the time period list in the channel's metadata, and the main service caches the metadata in memory, this operation only requires a memory query, allowing for sub-second returns even for long-duration recording retrievals.

[0082] The video storage service system of the present invention uses time periods as the smallest unit for locking / unlocking recordings. Therefore, when locking data within a certain time range, all time periods covered within that time range will be locked. Locking time periods directly affects the elimination of block files. If any time period on a block file is locked, the block file cannot be eliminated unless that time period is unlocked.

[0083] When the main center service receives a recording of a channel's time range that is locked / unlocked, it will compare the locked / unlocked time range with the time period list in the channel's metadata. It will set the status of all time periods that match the time range to locked or normal, and send a lock / unlock command to the node service where the time period is located. After receiving the lock / unlock command, the node service will set the status of the time period to locked or normal.

[0084] In the video storage system of this invention, recorded data is not permanently stored unless locked; it has a specific storage period. Each channel has its own storage period, and recorded data exceeding this period will be evicted and deleted. The smallest unit for evicting and deleting recorded data is a time period; all recorded data within the current time period must exceed its storage period before it is evicted. The video storage system of this invention evicts recorded data into time period evicting and block file evicting. The main center service is responsible for time period evicting. The main center service periodically traverses all channel information metadata in the system, monitoring and judging the difference between the end time and the current time in the time period list of each channel's information metadata, along with the storage period. If the end time of a time period exceeds the channel's storage period and the time period is not locked, a command to set the time period status to evicted is sent to the node service containing that time period. If the setting is successful, the time period is removed from the channel information metadata; if the setting fails, it will wait until the next check to try again until successful, at which point the time period is removed from the channel information metadata.

[0085] The block file metadata primarily describes information related to the storage resources allocated by the node service. In the video storage system of this invention, each node service exists as an independent node, managing only its allocated storage resources. Multiple logical block files are abstracted based on the size of the storage resources and the fixed block file size. These block files are numbered using an incrementing number. The node name and the block file number form the key of the block file metadata. This method ensures that the key of the block file metadata is unique within the video storage system of this invention. The value of the block file metadata includes basic information such as the node address, file number, and file size, as well as information related to the recording data and the corresponding storage resource type. The block file metadata is stored in the local node metadata database and is uniformly operated by the node service of the node. The block file metadata is mainly used for storage resource allocation management and data writing.

[0086] The storage space resources allocated to node services are managed through multiple fixed-size block files. The fixed size is configurable; for example, if the fixed size is 1GB, then the allocated storage space for a node service is 1TB, meaning the node service can have a maximum of 1024 block files for data storage. When a node service starts for the first time, it abstracts the logical block files managed by the node service based on the allocated storage space, calls the interface provided by the connected storage data layer to create physical storage units (files, objects, etc.), and generates block file information metadata, which is stored in the local node metadata database. Because the block file information metadata is relatively small, it is cached in memory throughout the node service's operation. Simultaneously, the total number of block files and the number of free block files (at this point, the number of free block files equals the total number of block files) are synchronized to the main service to update the node information metadata.

[0087] The video storage service system of this invention does not have an index file for storing metadata, but only data block files for storing data. The node service allocates unoccupied free block files to channels for data writing. A block file belongs to only one channel. Once a block file is allocated to a channel, it cannot be used by other channels until it is reclaimed as a free block. Therefore, if the load balancing mechanism is flawed, some channels may occupy unfilled block files for a long time, resulting in wasted space.

[0088] When a free block is allocated to a channel for data writing, the node service updates the block file information metadata, including the channel it belongs to, the start time, and other information. It then stores this metadata in a fixed space at the beginning of the block file and simultaneously synchronizes the channel information metadata with the main service. This approach of storing block file information metadata as block header information on the block file, along with the subsequent use of time period information metadata, allows for the reconstruction of all metadata even if it is lost or severely corrupted, ensuring the normal operation of the storage system and significantly improving its reliability.

[0089] In the video storage service system of this invention, the data stored will be cleared after the storage period is exceeded. The system clears the data by updating the metadata of the block file information for elimination and recycling. The data on the block file is not deleted. The block file storage resources are used in a cyclical manner. This implementation method can improve the performance of the storage system without wasting storage space resources.

[0090] The node service periodically checks the metadata of all block files it manages. If all time period information metadata contained in a block file is in the "erased" state, and there are currently no read / write operations on that block file, the node service sets all fields of that block file's metadata except for the basic information field to empty, sets the status field to "idle," updates the block file's metadata in its local node metadata database, deletes all time period information metadata contained in that block file's metadata, and simultaneously updates the metadata of the channel information that previously occupied that block file to the main central service. It should be noted that if a block file has locked time periods, then that block file cannot be evicted or reclaimed. Each time the node service successfully reclaims a block file, it increments the count of free block files and periodically updates the node information metadata to the main central service.

[0091] The time-segment information metadata mainly describes the attributes, type, location, and other related information of the stored video data, and is key to data access and management. In the video storage service system of this invention, the smallest unit of video data metadata is only the time segment; there is no metadata for frame data packets or GOP data packets, nor is there an index file for frame data. Only block files storing frame data exist, which significantly improves data read and write performance. Each time segment belongs to only one block file; that is, frame data for a given time segment is stored in only one block file. When a block file is full of data from a certain channel, and data needs to be written to a new block file, the data in the new block file must belong to a new time segment. Due to changes in other conditions, a block file may contain multiple time segments. All time segments in a block file can be numbered using an incrementing method. For example, if the time segment value of the metadata for a block file is 3, then these three time segments are numbered 1, 2, and 3. The node service uses the node name, block file number, and time period number to form the key of the time period information metadata. This method ensures that the key of the time period information metadata is unique within the video storage service system of this invention. In addition to containing basic information about the data within that time range, the value of the time period information metadata also records a list of keyframes within that time period to avoid performance degradation caused by multiple jumps to the data block file during data reading. The node service only needs to load the time period information metadata from the local node metadata database into memory when it begins reading data for a certain time period. Subsequently, it only needs to perform one I / O operation on the block file to read one or more GOP packet data. The time period information metadata is stored in the local node metadata database and is uniformly operated by the node service of the node. The time period information metadata is mainly used for processing related to video data read and write operations.

[0092] When a node service writes data to a channel, it will generate new time-segment metadata in the following situations: 1. When writing data to a new block file, the node service will generate new time-segment metadata. If this is not the first time data has been written to this node service, the previous time-segment metadata for that channel will be terminated. 2. When writing data to the block file being written to, if the time interval between the current frame data time and the previous frame data in that block file exceeds the maximum allowed time interval threshold (default is 10 seconds), the node service will generate new time-segment metadata and terminate the previous time-segment metadata. 3. When writing data to the block file being written to, if the current frame data time is earlier than the previous frame data in that block file (3 seconds), the node service will generate new time-segment metadata and terminate the previous time-segment metadata. This scenario is generally caused by server time synchronization, avoiding the problem of new data overwriting old data within the same time range.

[0093] When a node service generates new metadata for a new time period, it synchronizes the time period ID, start time, end time (at this point, the start time and end time are equal), recording type, and other information of the metadata with the main service so that the main service can update the channel information metadata of the channel to which the current time period belongs. Simultaneously, if it needs to end the metadata for the previous time period of a channel, it must synchronize the time period ID, start time, end time, recording type, and other information of the ended time period metadata with the main service so that the main service can update the channel information metadata of that channel. The smallest unit of data that the node service writes to the block file is a frame, which consists of a frame header and frame data. The frame header includes a tag string such as "slice", the time period ID of the frame, the frame time, the data length, and the frame identifier. The frame data is the actual data of the frame. The data structure on the block file starts with the block header, followed by appending data sequentially according to the format of the frame header and frame data. This implementation method can reconstruct all metadata even if it is lost or severely corrupted and unusable, ensuring the normal operation of the storage system and thus greatly improving the reliability of the storage system.

[0094] Users can directly read data for a specific time period from the node service via the SDK. Upon receiving the read command, the node service loads the time period information metadata from the local node metadata database into memory using the time period ID (this operation can be omitted if the time period information metadata is already in memory). Any subsequent operations to read data within that time period range only require comparing the start time of the read with the list of keyframes recorded in the metadata in memory to locate the offset of the frame to be read in the block file. Based on the value of this offset, one or more GOP packet data are continuously read from the block file, eliminating the need for further metadata I / O operations and greatly improving data reading performance.

[0095] The process of comparing the start frame time with the keyframe time to locate the data offset in the block file is as follows: 1. If the start frame time is equal to the start time of the time period information metadata, the data reading starts on the block file based on the offset address of the start time of that time period; 2. If the start time of the frame to be read is greater than the start time of the time period (this will not happen normally), the data reading starts on the file block based on the offset address of the start time of that time period; 3. If the start time of the frame to be read is less than the start time of the time period, the offset address of the keyframe closest to the start time of the frame to be read is found in the keyframe list of the time period information, and the data reading starts on the file block based on that offset address.

[0096] Figure 7 The flowchart for data writing provided by this invention is as follows: Figure 7 As shown, the key data writing process includes the following steps:

[0097] (1) The user calls the SDK to first create a channel with the main center service. If the channel has already been created, this operation can be omitted. (2) The user calls the SDK to write data. When the SDK starts writing data or determines that the current block is full, it requests a write node from the main center service. (3) The user calls the SDK to directly interact with the node service of the write node allocated by the main center service to continuously write data streams. (4) After receiving the written data, the node service determines whether a new block file needs to be selected for writing. If a new block file needs to be selected, operation (5) is executed. If a new block file does not need to be selected, operation (6) is executed. (5) The node service (6) The node service selects a block file from the list of free block files, updates the block file metadata, generates new time period information metadata, and periodically synchronizes the metadata with the main center service; (7) The node service selects the block file currently occupied by the channel, determines whether new time period information metadata needs to be generated, generates new time period information metadata if needed, updates the current time period information metadata if not needed, and periodically synchronizes the metadata with the main center service; (8) The node service successfully writes the data into the block file and updates the block file information and time period information metadata in the database; (9) Repeatedly execute operations (3), (4), (5), (6), and (7).

[0098] Figure 8 The flowchart for data reading provided by this invention is as follows: Figure 8 As shown, the key data reading process includes: (1) The user calls the SDK to enumerate channels with the main center service. If the user knows the channel to be read, this operation can be omitted; (2) The user calls the SDK to search the main center service for video recordings; (3) The main center service queries all time period lists in the metadata of the channel information according to the search conditions and returns all time period lists that meet the conditions; (4) The user calls the SDK to directly read data from the node service where a certain time period is located; (5) After receiving the data reading instruction, the node service will load the metadata of the time period information from the database into memory through the time period ID; (6) The node service compares the start time of the data to be read with the key frame list in the metadata of the time period information to locate the offset of the data to be read on the block file; (7) The node service locates the storage location of the data according to the offset and continuously reads the data; (8) Operations (4), (5), (6), and (7) are executed in a loop until all data is read.

[0099] Figure 9 The video recording lock / unlock flowchart provided by this invention is as follows: Figure 9As shown, the key process of video recording lock / unlock is as follows: (1) The user calls the SDK to lock or unlock a certain channel of video recording to the main center service; (2) The main center service retrieves the channel information metadata of the channel in memory according to the locked time range to find all time periods covered by the time range; (3) The main center service sends a lock / unlock instruction to the node service where the time period meets the condition; (4) After receiving the lock / unlock instruction for a certain time period, the node service updates the status field value of the time period information metadata according to the time period ID; (5) After the main center service successfully performs the lock / unlock operation on all the node services where the time period meets the condition, it will update the channel information metadata of the channel; (6) The operations (3), (4), and (5) are executed in a loop, and the operation success result is returned after completion.

[0100] Figure 10 The data elimination flowchart provided by this invention is referenced. Figure 10 The metadata detection and updating of this invention are described.

[0101] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: periodically detecting all channel information metadata through a central service, and detecting the time period list of each channel information metadata to determine whether the time period status is locked; in response to the time period status not being locked and the central service detecting that the end time of a time period exceeds the channel's storage period, sending an instruction to the node service where the time period is located to evict the time period through the central service; in response to the node service receiving the instruction, updating the status of the time period information metadata according to the time period identifier; and clearing the time period information from the channel information metadata and updating the channel information metadata to the database.

[0102] The main center service performs channel information metadata detection and update operations: (1) The main center service periodically detects all channel information metadata and checks and judges the time period list of each channel information metadata; (2) The main center service judges whether the time period status is locked. If it is locked, it executes (1). If it is not locked, it executes the next step; (3) When the main center service detects that the end time of a certain time period exceeds the storage period of the channel, the main center service sends an elimination instruction to the node service where the time period is located; (4) After receiving the elimination instruction for a certain time period, the node service updates the status of the information metadata of the time period according to the ID of the time period; (5) After the main center service successfully performs the elimination operation to the node service where the time period is located, it removes the information of the time period from the channel information metadata and updates the metadata of the channel information to the central database, and continues to perform the above operations in a loop.

[0103] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: periodically detecting all block file information metadata and time period information metadata through the node service to determine whether the current block file is currently undergoing read or write operations; in response to the current block file not undergoing read or write operations, detecting the status of all time period information metadata on the block file through the node service and determining whether the status of all time period information metadata is in an obsolete state; in response to the time period information metadata being in an obsolete state, reclaiming the block file as an idle block through the node service and deleting all time period information metadata on the block file; and updating the reclaimed block file information metadata through the node service and simultaneously synchronizing the metadata information with the central service.

[0104] The node service performs the detection and update operations of block file information and time period information metadata: (1) The node service periodically detects all block file information metadata and time period information metadata. If the current block file is being read or written, the next block file information metadata is detected; otherwise, the next step is executed. (2) The node service detects the status of all time period information metadata on a block file. If the status of a time period information metadata is locked, (1) is executed; otherwise, the next step is executed. (3) The node service detects that all time period information metadata on a block file is in the eviction state and executes the next step; otherwise, (1) is executed. (4) The node service reclaims the block file as an idle block and deletes all time period information metadata on the block file. (5) The node service updates the reclaimed block file information metadata and simultaneously synchronizes the relevant metadata information to the main center service. (6) The above operations are executed repeatedly.

[0105] Figure 11 The flowchart for metadata reconstruction provided by this invention is as follows: Figure 11As shown, the key process of metadata reconstruction includes: (1) The node service calculates the number of block files based on the currently allocated storage resources; (2) Generate the KEY of all block file information metadata, i.e., the block file ID, based on the node name and the incrementing number; (3) The node service iterates through all block file data and determines whether the traversal of all block files is complete. If not, proceed to the next step; if complete, end the traversal operation; (4) The node service reads the block header data of the block file based on the block file ID and generates block file information metadata; (5) The node service reads the frame header of the first frame on the block file and generates time period information metadata; (6) The node service stores the generated block file information and time period information metadata in the local node metadata database; (7) After the metadata is successfully stored, the node service synchronizes the information of the channel to which the block file belongs to the main center service; (8) The main center service generates or updates the channel information metadata of the channel based on the synchronized relevant information and stores or updates the channel information metadata in the central metadata database. After successful storage, it returns... (9) The node service continues to read the frame header information on the block file. If the current frame is a key frame, the timestamp and offset address of the frame are added to the key frame list of the information metadata of the time period and stored in the local node metadata database. If it is not a key frame, the node service continues to read the frame header of the next frame. (10) The node service continues to read the frame header information on the block file. If the current frame belongs to a new time period, the node service generates new time period information metadata and stores the newly generated time period information in the local node metadata database. After the metadata is successfully stored, the node service will synchronize the metadata of the previous end time period and the newly generated time period information metadata with the main center service. (11) The main center service updates the channel information metadata according to the metadata information synchronized by the node service and updates the channel information metadata in the central metadata database. (12) The node service determines whether the data reading of the current block file is completed. If it is completed, the node service continues to execute from (4). Otherwise, the node service continues to execute from (9). (13) The node service executes the above operations in a loop until all block files are traversed and read.

[0106] Different types of metadata are stored in different metadata databases, and different redundancy protection strategies are used for different metadata databases to ensure the consistency of metadata.

[0107] In some implementations, storing different types of metadata in different metadata databases and employing different redundancy protection strategies for these databases to ensure metadata consistency includes: using a clustered database to store the global metadata and using a dual-backup database to store the local node metadata. This provides redundant protection for all metadata and allows for the reconstruction of all metadata in the event of loss or severe corruption, ensuring the normal operation of the storage system.

[0108] The present invention has the following beneficial effects:

[0109] 1. Performance: Video retrieval only requires the main center service to retrieve channel information metadata in memory. Retrieval over a long period of time can also achieve a return time in seconds. Data reading and writing is a direct interaction with the node service, without any intermediate links or bottlenecks. In most cases, only one GOP data packet IO operation is performed, without IO operations for metadata indexes such as block files and time period information, which greatly improves data reading and writing performance.

[0110] 2. Reliability: Global metadata is stored using a clustered database, while local node metadata is stored using a dual-backup database. All metadata is redundantly protected, and in the event that metadata is lost or severely damaged and unusable, all metadata can be reconstructed to ensure the normal operation of the storage system.

[0111] 3. Scalability: The metadata in the storage system is distributed across various nodes in the system, and the storage space resources are uniformly and evenly allocated by the main central service, enabling the addition and deletion of nodes without interrupting business operations.

[0112] 4. Low cost, because each node in the system cluster is itself a storage system, supporting all the functional features of the system and capable of handling all business, without the need for additional costs such as providing metadata servers or management servers.

[0113] It should be particularly noted that the steps in the various embodiments of the above-described method for generating and storing metadata can be interleaved, substituted, added, or deleted. Therefore, these reasonable permutations and combinations of the method for generating and storing metadata should also fall within the protection scope of this invention, and the protection scope of this invention should not be limited to the embodiments.

[0114] Based on the above objectives, a second aspect of this invention proposes a system for generating and storing metadata. For example... Figure 12 As shown, system 200 includes the following modules: a definition module, used to define various types of metadata and set the association between different types of metadata; a management module, used to generate different types of metadata according to the definition and manage the corresponding types of metadata through different service processes; and a storage module, used to store different types of metadata in different metadata databases and adopt different redundancy protection strategies for different metadata databases to ensure the consistency of metadata.

[0115] In some implementations, the definition module is further configured to: divide metadata into node information, channel information, block file information, and time period information, wherein the node information is used to describe node service information, the channel information is used to describe camera recording information, the block file information is used to describe the storage resource information allocated by the node service, and the time period information is used to describe the attributes, type, and location of the video data.

[0116] In some implementations, the definition module is further configured to: set one channel information to correspond to multiple block file information, and set one block file information to correspond to multiple time period information.

[0117] In some implementations, the management module is further configured to: set node information and channel information as global metadata, and set block file information and time period information as local node metadata; and manage the global metadata through a central service, and manage the local node metadata through a node service.

[0118] In some implementations, the management module is further configured to: periodically detect all channel information metadata through a central service, and detect the time period list of each channel information metadata to determine whether the time period is locked; in response to the time period being unlocked and the central service detecting that the end time of a time period exceeds the storage period of the channel, send an instruction to the node service where the time period is located to eliminate the time period through the central service; in response to the node service receiving the instruction, update the status of the time period information metadata according to the time period identifier; and clear the time period information from the channel information metadata and update the channel information metadata in the database.

[0119] In some implementations, the management module is further configured to: periodically detect all block file information metadata and time period information metadata through the node service to determine whether the current block file is currently undergoing read or write operations; in response to the current block file not undergoing read or write operations, detect the status of all time period information metadata on the block file through the node service, and determine whether the status of all time period information metadata is in an obsolete state; in response to the time period information metadata being in an obsolete state, reclaim the block file as an idle block through the node service, and delete all time period information metadata on the block file; and update the reclaimed block file information metadata through the node service, and simultaneously synchronize metadata information to the central service.

[0120] In some implementations, the storage module is further configured to: store the global metadata using a cluster-mode database, and store the local node metadata using a dual-backup database.

[0121] Based on the above objectives, a third aspect of the present invention provides an electronic device, comprising: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executed by the processor to perform the following steps: S1, defining multiple types of metadata and setting associations between different types of metadata; S2, generating different types of metadata according to the definitions and managing the corresponding types of metadata through different service processes; and S3, storing different types of metadata in different metadata databases and employing different redundancy protection strategies for the different metadata databases to ensure the consistency of metadata.

[0122] In some implementations, the various types of metadata defined include: dividing metadata into node information, channel information, block file information, and time period information. The node information is used to describe node service information, the channel information is used to describe camera recording information, the block file information is used to describe the storage resource information allocated by the node service, and the time period information is used to describe the attributes, type, and location of the video data.

[0123] In some implementations, setting the association between different types of metadata includes: setting one channel information to correspond to multiple block file information, and setting one block file information to correspond to multiple time period information.

[0124] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: setting node information and channel information as global metadata, and setting block file information and time period information as local node metadata; and managing the global metadata through a central service and managing the local node metadata through a node service.

[0125] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: periodically detecting all channel information metadata through a central service, and detecting the time period list of each channel information metadata to determine whether the time period status is locked; in response to the time period status not being locked and the central service detecting that the end time of a time period exceeds the channel's storage period, sending an instruction to the node service where the time period is located to evict the time period through the central service; in response to the node service receiving the instruction, updating the status of the time period information metadata according to the time period identifier; and clearing the time period information from the channel information metadata and updating the channel information metadata to the database.

[0126] In some implementations, the step of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes includes: periodically detecting all block file information metadata and time period information metadata through the node service to determine whether the current block file is currently undergoing read or write operations; in response to the current block file not undergoing read or write operations, detecting the status of all time period information metadata on the block file through the node service and determining whether the status of all time period information metadata is in an obsolete state; in response to the time period information metadata being in an obsolete state, reclaiming the block file as an idle block through the node service and deleting all time period information metadata on the block file; and updating the reclaimed block file information metadata through the node service and simultaneously synchronizing the metadata information with the central service.

[0127] In some implementations, storing different types of metadata in different metadata databases and employing different redundancy protection strategies for different metadata databases to ensure metadata consistency includes: using a clustered database to store the global metadata and using a dual-backup database to store the local node metadata.

[0128] like Figure 13 The diagram shown is a hardware structure schematic of an embodiment of the electronic device for generating and storing metadata provided by the present invention.

[0129] For example Figure 13 The device shown is an example, which includes a processor 301 and a memory 302.

[0130] Processor 301 and memory 302 can be connected via a bus or other means. Figure 13 Taking the example of a connection between China and Israel via a bus.

[0131] The memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for generating and storing metadata in the embodiments of this application. The processor 301 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 302, thereby implementing the method for generating and storing metadata.

[0132] Memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of methods for generating and storing metadata, etc. Furthermore, memory 302 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 302 may optionally include memory remotely located relative to processor 301, and these remote memories may be connected to the local module via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0133] One or more methods for generating and storing metadata are stored in memory 302. When executed by processor 301, the method for generating and storing metadata in any of the above method embodiments is executed.

[0134] Any embodiment of an electronic device that performs the above-described method for generating and storing metadata can achieve the same or similar effects as any of the aforementioned method embodiments.

[0135] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a method for generating and storing metadata.

[0136] like Figure 14 The diagram shown is a schematic representation of an embodiment of the computer storage medium for generating and storing metadata provided by the present invention. Figure 14 Taking the computer storage medium shown as an example, the computer-readable storage medium 401 stores a computer program 402 that, when executed by a processor, performs the above method.

[0137] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for generating and storing metadata can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0138] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0139] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0140] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0141] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0142] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for generating and storing metadata, characterized in that, Includes the following steps: Define various types of metadata and set the relationships between different types of metadata; Different types of metadata are generated according to the definition, and the corresponding types of metadata are managed through different service processes, including: periodically detecting all channel information metadata through the central service, and detecting the time period list of each channel information metadata to determine whether the time period is locked. In response to a time period being unlocked and the central service detecting that the end time of a time period exceeds the storage period of the channel, the central service sends an instruction to the node service where the time period is located to evict the time period. In response to the node service receiving the instruction, the status of updating the time period information metadata according to the time period identifier; and The time period information is removed from the channel information metadata, and the channel information metadata is updated in the database. The node service periodically checks all block file information metadata and time period information metadata to determine whether the current block file is being read or written. In response to the fact that no read or write operation has been performed on the current block file, the node service detects the status of all time period information metadata on the block file and determines whether the status of the time period information metadata is in the eviction state. In response to the fact that all time period information metadata is in an obsolete state, the node service reclaims the block file as a free block and deletes all time period information metadata from the block file; and The node service updates the metadata of the reclaimed block files and simultaneously synchronizes the metadata information with the central service; and Different types of metadata are stored in different metadata databases, and different redundancy protection strategies are used for different metadata databases to ensure the consistency of metadata. The various types of metadata defined include: The metadata is divided into node information, channel information, block file information, and time period information. The node information is used to describe node service information, the channel information is used to describe camera recording information, the block file information is used to describe the storage resource information allocated by the node service, and the time period information is used to describe the attributes, type, and location of the video data. The node information metadata is stored in the central metadata database cluster and is uniformly operated by the main central service. When the main central service runs, it loads the node information metadata from the central database into memory. During the operation, it periodically persists and updates the node information in memory to the central metadata database cluster. The node information metadata is used for node service status monitoring and storage resource load balancing.

2. The method for generating and storing metadata according to claim 1, characterized in that, The settings for the association between different types of metadata include: Set one channel information to correspond to multiple block file information, and set one block file information to correspond to multiple time period information.

3. The method for generating and storing metadata according to claim 1, characterized in that, The steps of generating different types of metadata according to the definition and managing the corresponding types of metadata through different service processes include: Set node information and channel information as global metadata, and set block file information and time period information as local node metadata; and The global metadata is managed through a central service, and the local node metadata is managed through a node service.

4. The method for generating and storing metadata according to claim 3, characterized in that, The method of storing different types of metadata in different metadata databases and employing different redundancy protection strategies for different metadata databases to ensure metadata consistency includes: The global metadata is stored using a clustered database, and the local node metadata is stored using a dual-backup database.

5. A system for generating and storing metadata, characterized in that, include: The definition module is used to define various types of metadata and set the relationships between different types of metadata; The management module is used to generate different types of metadata according to the definition and manage the corresponding types of metadata through different service processes; as well as The storage module is used to store different types of metadata into different metadata databases and to employ different redundancy protection strategies for different metadata databases to ensure the consistency of metadata. The management module is also used to: periodically detect all channel information metadata through the central service, and detect the time period list of each channel information metadata to determine whether the time period is locked; In response to a time period being unlocked and the central service detecting that the end time of a time period exceeds the storage period of the channel, the central service sends an instruction to the node service where the time period is located to evict the time period. In response to the node service receiving the instruction, the status of the time period information metadata is updated according to the time period identifier; as well as The time period information is removed from the channel information metadata, and the channel information metadata is updated in the database. The node service periodically checks all block file information metadata and time period information metadata to determine whether the current block file is being read or written. In response to the fact that no read or write operation has been performed on the current block file, the node service detects the status of all time period information metadata on the block file and determines whether the status of the time period information metadata is in the eviction state. In response to the fact that all time period information metadata is in an obsolete state, the block file is reclaimed as a free block through the node service, and all time period information metadata on the block file is deleted; as well as The node service updates the metadata of the recycled block file information and simultaneously synchronizes the metadata information with the central service. The various types of metadata defined include: The metadata is divided into node information, channel information, block file information, and time period information. The node information is used to describe node service information, the channel information is used to describe camera recording information, the block file information is used to describe the storage resource information allocated by the node service, and the time period information is used to describe the attributes, type, and location of the video data. The node information metadata is stored in the central metadata database cluster and is uniformly operated by the main central service. When the main central service runs, it loads the node information metadata from the central database into memory. During the operation, it periodically persists and updates the node information in memory to the central metadata database cluster. The node information metadata is used for node service status monitoring and storage resource load balancing.

6. An electronic device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-4.

Images