Smart park articulated naturality web node data distributed storage system

Abstract

The invention discloses a smart park articulated naturality web node data distributed storage system, and relates to the technical field of smart park data storage. Comprising a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security guarantee module, a data verification module, a data adjustment module and an application interface layer, and all the modules cooperate to realize efficient storage, quality control and management of articulated naturality web node data. Aiming at a smart park articulated naturality web data storage pain point, efficient storage and management and control are realized through multi-module cooperation, a distributed architecture, mixed storage and intelligent scheduling are adopted, the storage efficiency and the resource utilization rate are improved, multi-copy backup, cross-physical deployment and automatic fault repair are realized, data reliability is guaranteed, and the system is suitable for large-scale popularization and application. The method comprises the steps of full-process encryption, authority control and multi-dimensional verification, building of a data security defense line, data dynamic optimization and standardization of an interface, enhancement of data availability and system compatibility, and adaptation to the dynamic increase demand of park data.

Description

Technical Field

[0001] This invention relates to the field of smart park data storage technology, and in particular to a distributed storage system for smart park video network node data. Background Technology

[0002] As a crucial component of smart city construction, smart parks integrate a large number of cameras, sensors, and other devices into their video network systems. This enables intelligent management across multiple scenarios, including security monitoring, personnel scheduling, and equipment maintenance. The video streams, image data, and associated sensor data generated by video network nodes are characterized by massive data volumes, high real-time requirements, and long lifecycles. This places stringent demands on the performance, reliability, and scalability of the storage system. Existing smart park video network data storage solutions primarily employ centralized or simple distributed storage architectures, which suffer from the following technical drawbacks: Firstly, in a centralized storage architecture, all data is stored centrally on the core server, resulting in high data access response latency. This fails to meet the real-time storage and concurrent access requirements of multiple video streams and poses a single point of failure risk. First, a core server failure will paralyze the entire storage system. Second, traditional distributed storage architectures have fixed node configurations, requiring extensive hardware modifications and system reconstruction for expansion, resulting in poor flexibility and difficulty in adapting to the dynamic growth in the number of video network devices and data volume in the park. Third, data backup mechanisms are simplistic, often employing a local dual-copy model, which can easily lead to data loss and insufficient reliability in the event of regional failures (such as power outages or fires in the data center). Fourth, the lack of intelligent scheduling mechanisms in storage resource allocation leads to uneven load distribution among nodes, with some nodes operating at high loads while others remain idle, resulting in low storage resource utilization. Fifth, the lack of robust security protection mechanisms during data transmission and storage means that the video network data in the park contains a large amount of sensitive information, making it susceptible to data leakage or tampering. Therefore, developing a distributed storage system with low latency, high reliability, elastic scalability, and intelligent scheduling capabilities has become a pressing technical problem to be solved in the construction of smart park video networks. Summary of the Invention

[0003] The smart park video network node data distributed storage system proposed in this invention solves the above-mentioned shortcomings of the prior art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: The distributed storage system for video network node data in the smart park includes a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security module, a data verification module, a data adjustment module, and an application interface layer. These modules work together to achieve efficient storage, quality control, and management of video network node data. Each module is further subdivided into units, and the specific structure is as follows. The data acquisition layer serves as the system's data input source, connecting to various video network devices within the park to complete data acquisition, preprocessing, and standardized packaging, providing a standardized data foundation for subsequent storage. The distributed storage node cluster is the core carrier of the system's data storage. It adopts a unitized architecture and a hybrid storage mode, and achieves efficient and reliable data storage through data sharding, multi-replica backup and elastic expansion mechanisms. The metadata management module undertakes the key function of data location. Through a distributed architecture and efficient indexing mechanism, it enables rapid retrieval and real-time synchronization of metadata, ensuring that data location information is accurate and verifiable. The intelligent scheduling module is the central hub for system resource scheduling. Based on multi-dimensional information such as node load, network bandwidth, and data popularity, it dynamically allocates storage tasks and adjusts resource configuration to ensure the overall operating efficiency of the system.

[0005] Furthermore, the data security module constructs a full lifecycle security protection system, preventing data leakage, tampering, and damage risks through encryption, access control, and verification mechanisms from data transmission and storage to access. The data verification module is responsible for ensuring data quality. Through multi-dimensional verification units, it comprehensively verifies the format, integrity, consistency, and authenticity of the collected and stored data, and filters out abnormal data. The data adjustment module dynamically optimizes the data based on the data verification results and application requirements, including format correction, data completion, quality improvement and redundancy compression, to improve data availability and storage efficiency. The application interface layer serves as an interaction bridge between the system and upper-layer applications, providing standardized interfaces and development tools to support data query, read, and export operations, and enabling seamless integration with various management systems within the park.

[0006] Furthermore, the data acquisition layer includes: Multiple data acquisition terminals: establish communication connections with video network equipment within the park, support multiple mainstream video transmission protocols such as RTSP, RTMP, and GB28181, and realize real-time acquisition of video streams and associated sensing data; Preprocessing unit: Built-in format conversion, redundancy filtering, keyframe extraction, and metadata encapsulation functions. It performs format standardization conversion, redundancy filtering, and keyframe extraction on the acquired video stream data. At the same time, it extracts and encapsulates metadata related to the device number, acquisition time, and location information of the video stream, forming standardized data packets, which are then transmitted to the distributed storage node cluster. FFmpeg is used to uniformly convert the video format to H.265 format. The keyframe extraction interval is 5-15 frames. Redundant data of blurry and repeated frames is filtered out, and metadata of device number, acquisition time, and location information is extracted.

[0007] Furthermore, the distributed storage node cluster consists of multiple distributed storage nodes, including: Node orchestration unit: Based on the Kubernetes orchestration engine, it is responsible for the creation, startup, shutdown, scaling up / down and health status monitoring of distributed storage node container instances. It collects the node hardware resource status (CPU, memory, storage) in real time and feeds it back to the intelligent scheduling module. It receives task allocation instructions to realize dynamic resource scheduling and load balancing. When a node failure is detected, it automatically triggers the isolation and replacement mechanism. Sharded storage unit: Built-in sharding strategy engine and hash allocation sub-unit, supporting configurable shard size from 128KB to 512KB, adaptively adjusting granularity according to data type (256KB by default for video stream), calculating data shard hash value through consistent hashing algorithm, and achieving uniform distribution of data shards by combining node load and network topology. The data sharding hash allocation algorithm is as follows: ; in, Number the target storage node for data sharding; To use the SHA-256 hash function, output a fixed-length hash value; A unique identifier for standardized data packets (generated by device number + collection timestamp); The index of the data shard in the current packets (incrementing from 0); This represents the total number of nodes in the distributed storage node cluster. Constraints: The data fragment size range is [range missing]. (Default value is 256KB); The replica management unit includes replica creation, distribution, synchronization, and repair sub-units. It creates 3-5 replicas based on the importance level of the data (5 for core security videos and 3 for general perception data). It uses a cross-physical location deployment strategy to distribute replicas to different racks or data center nodes. It ensures the consistency of primary and secondary data through an incremental synchronization mechanism. When a replica is detected to be damaged or lost, it automatically repairs it or triggers a re-storage process. The formula for selecting replica storage nodes is as follows: ; in, Let i be the storage node number of the i-th replica; i is the replica number (from 1 to K-1, where K is the total number of replicas). The node interval step size (values ​​are...) (To ensure copies are distributed) The number of replicas (value range 3-5, default 3); Constraints: Corresponding nodes and The nodes are not in the same physical rack; Hybrid storage unit: integrates SSD storage sub-unit, HDD storage sub-unit and storage resource monitoring sub-unit. It receives the heat analysis results of the intelligent scheduling module through the storage media scheduling sub-unit to realize the tiered storage of hot data (access frequency > 10 times / minute) and cold data (access frequency < 1 time / hour). It monitors the status of storage media in real time and sends expansion warning when the storage space utilization exceeds 80%. Cluster communication unit: It adopts a gigabit Ethernet + fiber optic redundant communication architecture, including a data transmission subunit and a status synchronization subunit. It realizes data transmission, replica synchronization and command interaction through the TCP / IP protocol suite, supports dynamic adjustment of transmission rate, and realizes real-time synchronization of node status through a heartbeat mechanism every 2 seconds. Storage unit: It adopts a hybrid storage architecture of SSD and HDD to meet the storage needs of hot data and cold data respectively; Computing Unit: Supports the computing tasks of data sharding and replica management within the node.

[0008] Network Unit: Ensures network communication between nodes and other nodes within the cluster as well as external modules.

[0009] Furthermore, the metadata management module adopts a distributed metadata server cluster architecture, which includes: Metadata server cluster: It adopts a master-slave architecture, with the master server responsible for writing metadata and the slave server responsible for reading and backup. It stores metadata information such as data shard identifier, storage node address, number of replicas, data size, access time and data type. Metadata index building unit: It adopts a B+ tree index structure to achieve fast retrieval of metadata and supports real-time synchronous updates of metadata.

[0010] Furthermore, the intelligent scheduling module is communicatively connected to the data acquisition layer, the distributed storage node cluster, and the metadata management module, and includes: Load monitoring unit: Collects CPU utilization, memory usage and storage space utilization of each distributed storage node every 3-10 seconds. When CPU utilization exceeds 70% or storage space utilization exceeds 80%, the load balancing mechanism is triggered. Bandwidth monitoring unit: Real-time monitoring of network transmission bandwidth and latency between nodes; adjusts data transmission path when transmission bandwidth is below 100Mbps. Heat Analysis Unit: Based on data access frequency and access time, the data is divided into hot data, warm data and cold data, providing a basis for hierarchical storage; The formula for determining data popularity is as follows: Data access popularity calculation model Core logic: Based on the data access frequency per unit time, classify data into hot, warm, and cold data; The formula expression is: ; in, Data access frequency (unit: times / minute); For statistical period Total number of data accesses within; The statistical period is specified (default value is 1 minute). Judgment rules: Hot data: times / minute (stored on SSD); Temperature data: (Stored in a mixed SSD and HDD area); Cold data: Times / hour (stored in HDD); Task allocation unit: Based on load monitoring, bandwidth monitoring and heat analysis results, dynamically adjust the allocation of data shard storage nodes, replica distribution and data storage media to achieve load balancing and storage efficiency optimization; The load balancing trigger determination formula is as follows: The core logic of the node load comprehensive evaluation index is to combine CPU utilization, memory usage, and storage space utilization to calculate the node load comprehensive index and trigger the load balancing mechanism. Formula expression: ; in, The node load comprehensive index (value range 0-1). Node CPU utilization (value range 0-1); This represents the node's memory usage (value range 0-1). This represents the node's storage space utilization rate (value range: 0-1). Weighting coefficients: , , (satisfy ); Triggering conditions: when or At that time, load balancing is triggered directly; when When necessary, activate load warning and dynamically adjust based on bandwidth status; Formula for determining network bandwidth threshold: Core logic: Monitor the transmission bandwidth between nodes and adjust the data transmission path when it falls below a threshold; Formula expression: ; in, Minimum transmission bandwidth threshold (unit: Mbps); The total amount of data transmitted per unit time (unit: Mbps); This is the data redundancy factor (default value is 0.2, adapted to the requirements of copy transmission). The maximum allowable transmission delay (default value is 10ms); Triggering conditions: When the actual transmission bandwidth between nodes When the default threshold is 100Mbps, adjust the data transmission path.

[0011] Furthermore, the data security module includes: Encryption Unit: Employs the AES-256 encryption algorithm to encrypt data packets during transmission and data fragments in storage. The AES-256 encryption key derivation formula derives the data encryption key based on the master key and device identifier, ensuring the uniqueness of encryption during transmission and storage. The formula expression is as follows: ; in, AES-256 encryption key (256 bits); For key-based cryptographic derivation function 2; The system's global master key (pre-configured); A unique identifier for the data acquisition device; This represents the number of iterations (default value is 10000). This is the key length (value is 256 bits). Application scenario: Standardized data during the data transmission phase Encryption involves encrypting data in segments during the storage phase, and deriving the key using the same algorithm during decryption. Access Control Unit: Role-based access control policies assign granular data access permissions to different users, such as park administrators, maintenance personnel, and ordinary employees; Data verification unit: It adopts the CRC32 verification algorithm to perform periodic verification on the stored data fragments and copies every 24 hours. When data corruption or loss is detected, it triggers a copy repair or re-storage mechanism. The CRC32 checksum algorithm verifies data integrity and detects data corruption or tampering through a cyclic redundancy check algorithm. The calculation formula is as follows: ; in, It is a 32-bit cyclic redundancy check value; for Standard polynomial (value 0xEDB88320); The data to be verified is fragmented into byte streams; This is a bitwise XOR operation; This is a left shift operation; Verification logic: Calculate and save data fragments during storage. When reading, the data is recalculated and compared. If the two are inconsistent, the data is considered corrupted, triggering the copy repair mechanism.

[0012] Furthermore, the data verification module employs a multi-dimensional verification unit deployed in a modular fashion, comprising: Format verification unit: verifies the format standardization of collected and stored data to ensure compliance with preset standardized format requirements; Integrity verification unit: detects whether there are integrity issues such as missing fields and data truncation in the data; Consistency verification unit: verifies the data consistency of the same data shard and its replicas to avoid data synchronization anomalies; Authenticity verification unit: Verifies the authenticity of the data source by comparing the device identification and signature information of the data source, and prevents the injection of false data.

[0013] Furthermore, the data adjustment module adopts a dynamically adjusted unit with modular deployment, which includes: Format Correction Unit: Receives abnormal feedback from the format verification unit and automatically corrects non-standard format data; Data completion unit: For missing data detected by the integrity verification unit, it performs reasonable completion or marks the missing status based on historical data patterns and related data; Quality Optimization Unit: Adjusts data resolution and encoding parameters according to application scenario requirements (such as security monitoring and data analysis) to improve the quality of data application; Redundant compression unit: The LZ4 compression algorithm is used to redundantly compress the verified valid data to reduce storage usage.

[0014] Furthermore, the application interface layer provides RESTful API interfaces and SDK development toolkits, supporting integration with upper-level application systems such as park security management systems and operation and maintenance management platforms to realize data query, read, export, and delete operations.

[0015] Compared with existing technologies, the beneficial effects of this invention are: 1. This invention supports multi-protocol data acquisition and standardized preprocessing, combined with configurable sharding and hybrid storage, hierarchical storage based on data popularity, and dynamic resource scheduling, which greatly improves storage and access efficiency. 2. This invention employs multi-replica deployment across physical locations, automatically isolates and replaces faulty nodes, and repairs corrupted data in real time, thus avoiding the risk of data loss caused by single-point failures and regional failures. 3. This invention features end-to-end AES-256 encryption, fine-grained access control, and periodic CRC32 verification to comprehensively prevent data leakage, tampering, and damage. Based on Kubernetes, it enables elastic scaling of nodes to adapt to the dynamic growth of campus equipment and data volume without the need for large-scale system reconstruction. 4. This invention filters abnormal data through multi-dimensional verification, dynamically optimizes data format, and completes missing information to improve data availability and application adaptability. At the same time, it provides standardized interfaces and development tools to seamlessly connect with various management systems in the park and support data application needs in multiple scenarios. In summary, this system addresses the pain points of data storage in smart park video network systems. It achieves efficient storage and management through multi-module collaboration, adopts a distributed architecture, hybrid storage, and intelligent scheduling to improve storage efficiency and resource utilization, ensures data reliability through multi-copy backup, cross-physical deployment, and automatic fault repair, strengthens data security through end-to-end encryption, access control, and multi-dimensional verification, and enhances data availability and system compatibility through dynamic data optimization and standardized interfaces to adapt to the dynamic growth needs of park data. Attached Figure Description

[0016] Figure 1 This is a flowchart of the overall system process of the smart park video network node data distributed storage system proposed in this invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0019] Example, refer to Figure 1 The smart park video network node data distributed storage system includes a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security module, a data verification module, a data adjustment module, and an application interface layer. The modules work together to achieve efficient storage, quality control and management of video network node data. Each module is subdivided into units, and the specific structure is as follows. The data acquisition layer serves as the system's data input source, connecting with various video network devices within the park to complete data acquisition, preprocessing, and standardized packaging, providing a standardized data foundation for subsequent storage. Distributed storage node clusters are the core carriers of system data storage. They adopt a unitized architecture and hybrid storage mode, and achieve efficient and reliable data storage through data sharding, multi-replica backup and elastic expansion mechanisms. The metadata management module undertakes the key function of data location. Through a distributed architecture and efficient indexing mechanism, it enables rapid retrieval and real-time synchronization of metadata, ensuring that data location information is accurate and verifiable. The intelligent scheduling module is the central hub of system resource scheduling. Based on multi-dimensional information such as node load, network bandwidth, and data popularity, it dynamically allocates storage tasks and adjusts resource configuration to ensure the overall operating efficiency of the system.

[0020] In this invention, the data security module constructs a full lifecycle security protection system, from data transmission and storage to access, and prevents the risks of data leakage, tampering and damage through encryption, access control and verification mechanisms; The data verification module is responsible for ensuring data quality. Through multi-dimensional verification units, it comprehensively verifies the format, integrity, consistency, and authenticity of the collected and stored data, and filters out abnormal data. The data adjustment module dynamically optimizes the data based on data verification results and application requirements, including format correction, data completion, quality improvement and redundancy compression, to improve data availability and storage efficiency. The application interface layer serves as an interaction bridge between the system and upper-layer applications. It provides standardized interfaces and development tools, supports data query, reading, and export operations, and enables seamless integration with various management systems in the park.

[0021] In this invention, the data acquisition layer includes: Multiple data acquisition terminals: establish communication connections with video network equipment within the park, support multiple mainstream video transmission protocols such as RTSP, RTMP, and GB28181, and realize real-time acquisition of video streams and associated sensing data; Preprocessing unit: Built-in format conversion, redundancy filtering, keyframe extraction, and metadata encapsulation functions. It performs format standardization conversion, redundancy filtering, and keyframe extraction on the acquired video stream data. At the same time, it extracts and encapsulates metadata related to the device number, acquisition time, and location information of the video stream, forming standardized data packets, which are then transmitted to the distributed storage node cluster. FFmpeg is used to uniformly convert the video format to H.265 format. The keyframe extraction interval is 5-15 frames. Redundant data of blurry and repeated frames is filtered out, and metadata of device number, acquisition time, and location information is extracted.

[0022] In this invention, the distributed storage node cluster consists of multiple distributed storage nodes, including: Node orchestration unit: Based on the Kubernetes orchestration engine, it is responsible for the creation, startup, shutdown, scaling up / down and health status monitoring of distributed storage node container instances. It collects the node hardware resource status (CPU, memory, storage) in real time and feeds it back to the intelligent scheduling module. It receives task allocation instructions to realize dynamic resource scheduling and load balancing. When a node failure is detected, it automatically triggers the isolation and replacement mechanism. Sharded storage unit: Built-in sharding strategy engine and hash allocation sub-unit, supporting configurable shard size from 128KB to 512KB, adaptively adjusting granularity according to data type (256KB by default for video stream), calculating data shard hash value through consistent hashing algorithm, and achieving uniform distribution of data shards by combining node load and network topology. The data sharding hash allocation algorithm is as follows: ; in, Number the target storage node for data sharding; To use the SHA-256 hash function, output a fixed-length hash value; A unique identifier for standardized data packets (generated by device number + collection timestamp); The index of the data shard in the current packets (incrementing from 0); This represents the total number of nodes in the distributed storage node cluster. Constraints: The data fragment size range is [range missing]. (Default value is 256KB); The replica management unit includes replica creation, distribution, synchronization, and repair sub-units. It creates 3-5 replicas based on the importance level of the data (5 for core security videos and 3 for general perception data). It uses a cross-physical location deployment strategy to distribute replicas to different racks or data center nodes. It ensures the consistency of primary and secondary data through an incremental synchronization mechanism. When a replica is detected to be damaged or lost, it automatically repairs it or triggers a re-storage process. The formula for selecting replica storage nodes is as follows: ; in, Let i be the storage node number of the i-th replica; i is the replica number (from 1 to K-1, where K is the total number of replicas). The node interval step size (values ​​are...) (To ensure copies are distributed) The number of replicas (value range 3-5, default 3); Constraints: Corresponding nodes and The nodes are not in the same physical rack; Hybrid storage unit: integrates SSD storage sub-unit, HDD storage sub-unit and storage resource monitoring sub-unit. It receives the heat analysis results of the intelligent scheduling module through the storage media scheduling sub-unit to realize the tiered storage of hot data (access frequency > 10 times / minute) and cold data (access frequency < 1 time / hour). It monitors the status of storage media in real time and sends expansion warning when the storage space utilization exceeds 80%. Cluster communication unit: It adopts a gigabit Ethernet + fiber optic redundant communication architecture, including a data transmission subunit and a status synchronization subunit. It realizes data transmission, replica synchronization and command interaction through the TCP / IP protocol suite, supports dynamic adjustment of transmission rate, and realizes real-time synchronization of node status through a heartbeat mechanism every 2 seconds. Storage unit: It adopts a hybrid storage architecture of SSD and HDD to meet the storage needs of hot data and cold data respectively; Computing Unit: Supports the computing tasks of data sharding and replica management within the node.

[0023] Network Unit: Ensures network communication between nodes and other nodes within the cluster as well as external modules.

[0024] In this invention, the metadata management module adopts a distributed metadata server cluster architecture, which includes: Metadata server cluster: It adopts a master-slave architecture, with the master server responsible for writing metadata and the slave server responsible for reading and backup. It stores metadata information such as data shard identifier, storage node address, number of replicas, data size, access time and data type. Metadata index building unit: It adopts a B+ tree index structure to achieve fast retrieval of metadata and supports real-time synchronous updates of metadata.

[0025] In this invention, the intelligent scheduling module is communicatively connected to the data acquisition layer, the distributed storage node cluster, and the metadata management module, and includes: Load monitoring unit: Collects CPU utilization, memory usage and storage space utilization of each distributed storage node every 3-10 seconds. When CPU utilization exceeds 70% or storage space utilization exceeds 80%, the load balancing mechanism is triggered. Bandwidth monitoring unit: Real-time monitoring of network transmission bandwidth and latency between nodes; adjusts data transmission path when transmission bandwidth is below 100Mbps. Heat Analysis Unit: Based on data access frequency and access time, the data is divided into hot data, warm data and cold data, providing a basis for hierarchical storage; The formula for determining data popularity is as follows: Data access popularity calculation model Core logic: Based on the data access frequency per unit time, classify data into hot, warm, and cold data; The formula expression is: ; in, Data access frequency (unit: times / minute); For statistical period Total number of data accesses within; The statistical period is specified (default value is 1 minute). Judgment rules: Hot data: times / minute (stored on SSD); Temperature data: (Stored in a mixed SSD and HDD area); Cold data: Times / hour (stored in HDD); Task allocation unit: Based on load monitoring, bandwidth monitoring and heat analysis results, dynamically adjust the allocation of data shard storage nodes, replica distribution and data storage media to achieve load balancing and storage efficiency optimization; The load balancing trigger determination formula is as follows: The core logic of the node load comprehensive evaluation index is to combine CPU utilization, memory usage, and storage space utilization to calculate the node load comprehensive index and trigger the load balancing mechanism. Formula expression: ; in, The node load comprehensive index (value range 0-1). Node CPU utilization (value range 0-1); This represents the node's memory usage (value range 0-1). This represents the node's storage space utilization rate (value range: 0-1). Weighting coefficients: , , (satisfy ); Triggering conditions: when or At that time, load balancing is triggered directly; when When necessary, activate load warning and dynamically adjust based on bandwidth status; Formula for determining network bandwidth threshold: Core logic: Monitor the transmission bandwidth between nodes and adjust the data transmission path when it falls below a threshold; Formula expression: ; in, Minimum transmission bandwidth threshold (unit: Mbps); The total amount of data transmitted per unit time (unit: Mbps); This is the data redundancy factor (default value is 0.2, adapted to the requirements of copy transmission). The maximum allowable transmission delay (default value is 10ms); Triggering conditions: When the actual transmission bandwidth between nodes When the default threshold is 100Mbps, adjust the data transmission path.

[0026] In this invention, the data security protection module includes: Encryption Unit: Employs the AES-256 encryption algorithm to encrypt data packets during transmission and data fragments in storage. The AES-256 encryption key derivation formula derives the data encryption key based on the master key and device identifier, ensuring the uniqueness of encryption during transmission and storage. The formula expression is as follows: ; in, AES-256 encryption key (256 bits); For key-based cryptographic derivation function 2; The system's global master key (pre-configured); A unique identifier for the data acquisition device; This represents the number of iterations (default value is 10000). This is the key length (value is 256 bits). Application scenario: Standardized data during the data transmission phase Encryption involves encrypting data in segments during the storage phase, and deriving the key using the same algorithm during decryption. Access Control Unit: Role-based access control policies assign granular data access permissions to different users, such as park administrators, maintenance personnel, and ordinary employees; Data verification unit: It adopts the CRC32 verification algorithm to perform periodic verification on the stored data fragments and copies every 24 hours. When data corruption or loss is detected, it triggers a copy repair or re-storage mechanism. The CRC32 checksum algorithm verifies data integrity and detects data corruption or tampering through a cyclic redundancy check algorithm. The calculation formula is as follows: ; in, It is a 32-bit cyclic redundancy check value; for Standard polynomial (value 0xEDB88320); The data to be verified is fragmented into byte streams; This is a bitwise XOR operation; This is a left shift operation; Verification logic: Calculate and save data fragments during storage. When reading, the data is recalculated and compared. If the two are inconsistent, the data is considered corrupted, triggering the copy repair mechanism.

[0027] In this invention, the data verification module employs a multi-dimensional verification unit deployed in a modular fashion, comprising: Format verification unit: verifies the format standardization of collected and stored data to ensure compliance with preset standardized format requirements; Integrity verification unit: detects whether there are integrity issues such as missing fields and data truncation in the data; Consistency verification unit: verifies the data consistency of the same data shard and its replicas to avoid data synchronization anomalies; Authenticity verification unit: Verifies the authenticity of the data source by comparing the device identification and signature information of the data source, and prevents the injection of false data.

[0028] In this invention, the data adjustment module adopts a dynamically adjusted unit with modular deployment, which includes: Format Correction Unit: Receives abnormal feedback from the format verification unit and automatically corrects non-standard format data; Data completion unit: For missing data detected by the integrity verification unit, it performs reasonable completion or marks the missing status based on historical data patterns and related data; Quality Optimization Unit: Adjusts data resolution and encoding parameters according to application scenario requirements (such as security monitoring and data analysis) to improve the quality of data application; Redundant compression unit: The LZ4 compression algorithm is used to redundantly compress the verified valid data to reduce storage usage.

[0029] In this invention, the application interface layer provides RESTful API interfaces and SDK development toolkits, supporting the connection with upper-level application systems such as park security management systems and operation and maintenance management platforms, and realizing data query, reading, export and deletion operations.

[0030] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A distributed data storage system for smart park video network nodes, characterized in that, It includes a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security module, a data verification module, a data adjustment module, and an application interface layer. These modules work together to achieve efficient storage, quality control, and management of video network node data. Each module is further subdivided into units, and the specific structure is as follows. The data acquisition layer is used to connect with various video network devices in the park, complete the data acquisition, preprocessing and standardized packaging, and provide a standardized data foundation for subsequent storage. The distributed storage node cluster adopts a unitized architecture and a hybrid storage mode, and achieves efficient and reliable data storage through data sharding, multi-replica backup and elastic expansion mechanisms; The metadata management module, through a distributed architecture and efficient indexing mechanism, enables rapid retrieval and real-time synchronization of metadata, ensuring that data location information is accurate and verifiable. The intelligent scheduling module dynamically allocates storage tasks and adjusts resource configurations based on multi-dimensional information such as node load, network bandwidth, and data popularity, ensuring the overall operating efficiency of the system.

2. The smart park video network node data distributed storage system according to claim 1, characterized in that, The data security module prevents data leakage, tampering, and damage risks through encryption, access control, and verification mechanisms from data transmission and storage to access. The data verification module uses a multi-dimensional verification unit to comprehensively verify the format, integrity, consistency, and authenticity of the collected and stored data, and filters out abnormal data. The data adjustment module dynamically optimizes the data based on the data verification results and application requirements, including format correction, data completion, quality improvement and redundancy compression, to improve data availability and storage efficiency. The application interface layer provides standardized interfaces and development tools, supports data query, read and export operations, and enables seamless integration with various management systems in the park.

3. The smart park video network node data distributed storage system according to claim 1, characterized in that, The data acquisition layer includes: Multiple data acquisition terminals: establish communication connections with video network equipment within the park, support multiple mainstream video transmission protocols such as RTSP, RTMP, and GB28181, and realize real-time acquisition of video streams and associated sensing data; Preprocessing unit: Built-in format conversion, redundancy filtering, keyframe extraction and metadata encapsulation functions. It performs format standardization conversion, redundancy filtering and keyframe extraction on the acquired video stream data. At the same time, it extracts and encapsulates the metadata of the device number, acquisition time and location information associated with the video stream, and transmits it to the distributed storage node cluster after forming standardized data packets.

4. The smart park video network node data distributed storage system according to claim 1, characterized in that, The distributed storage node cluster consists of multiple distributed storage nodes, including: Node orchestration unit: Based on the Kubernetes orchestration engine, it is responsible for the creation, startup, shutdown, scaling up / down and health status monitoring of distributed storage node container instances. It collects the node hardware resource status in real time and feeds it back to the intelligent scheduling module. It receives task allocation instructions to realize dynamic resource scheduling and load balancing. When a node failure is detected, it automatically triggers the isolation and replacement mechanism. Sharded storage unit: Built-in sharding strategy engine and hash allocation sub-unit, supports configurable shard size from 128KB to 512KB, adaptively adjusts granularity according to data type, calculates data shard hash value through consistent hashing algorithm, and achieves uniform distribution of data shards by combining node load and network topology. The replica management unit includes replica creation, distribution, synchronization, and repair sub-units. It creates 3-5 replicas based on the importance level of the data, and uses a cross-physical location deployment strategy to distribute replicas to different racks or data center nodes. It ensures the consistency of primary and secondary data through an incremental synchronization mechanism, and automatically repairs or triggers a re-storage process when replica damage or loss is detected. Hybrid storage unit: integrates SSD storage sub-unit, HDD storage sub-unit and storage resource monitoring sub-unit. It receives the heat analysis results of the intelligent scheduling module through the storage media scheduling sub-unit to realize the tiered storage of hot data and cold data, monitors the status of storage media in real time, and sends expansion warning when the storage space utilization exceeds 80%. Cluster communication unit: It adopts a gigabit Ethernet + fiber optic redundant communication architecture, including a data transmission subunit and a status synchronization subunit. It realizes data transmission, replica synchronization and command interaction through the TCP / IP protocol suite, supports dynamic adjustment of transmission rate, and realizes real-time synchronization of node status through a heartbeat mechanism every 2 seconds. Storage unit: It adopts a hybrid storage architecture of SSD and HDD to meet the storage needs of hot data and cold data respectively; Computing unit: Supports computing tasks for data sharding and replica management within a node; Network Unit: Ensures network communication between nodes and other nodes within the cluster as well as external modules.

5. The smart park video network node data distributed storage system according to claim 1, characterized in that, The metadata management module adopts a distributed metadata server cluster architecture, which includes: Metadata server cluster: It adopts a master-slave architecture, with the master server responsible for writing metadata and the slave server responsible for reading and backup. It stores metadata information such as data shard identifier, storage node address, number of replicas, data size, access time and data type. Metadata index building unit: It adopts a B+ tree index structure to achieve fast retrieval of metadata and supports real-time synchronous updates of metadata.

6. The smart park video network node data distributed storage system according to claim 1, characterized in that, The intelligent scheduling module is communicatively connected to the data acquisition layer, the distributed storage node cluster, and the metadata management module, and includes: Load monitoring unit: Collects CPU utilization, memory usage and storage space utilization of each distributed storage node every 3-10 seconds. When CPU utilization exceeds 70% or storage space utilization exceeds 80%, the load balancing mechanism is triggered. Bandwidth monitoring unit: Real-time monitoring of network transmission bandwidth and latency between nodes; adjusts data transmission path when transmission bandwidth is below 100Mbps. Heat Analysis Unit: Based on data access frequency and access time, the data is divided into hot data, warm data and cold data, providing a basis for hierarchical storage; Task allocation unit: Based on load monitoring, bandwidth monitoring and heat analysis results, dynamically adjust the allocation of data shard storage nodes, replica distribution and data storage media to achieve load balancing and storage efficiency optimization.

7. The smart park video network node data distributed storage system according to claim 2, characterized in that, The data security module includes: Encryption Unit: Employs the AES-256 encryption algorithm to encrypt data packets during transmission and data fragments in storage. Access Control Unit: Role-based access control policies assign granular data access permissions to different users, such as park administrators, maintenance personnel, and ordinary employees; Data verification unit: It adopts the CRC32 verification algorithm to perform periodic verification on the stored data fragments and copies every 24 hours. When data corruption or loss is detected, it triggers a copy repair or re-storage mechanism.

8. The smart park video network node data distributed storage system according to claim 1, characterized in that, The data verification module employs a multi-dimensional verification unit deployed in a modular fashion, which includes: Format verification unit: verifies the format standardization of collected and stored data to ensure compliance with preset standardized format requirements; Integrity verification unit: detects whether there are integrity issues such as missing fields and data truncation in the data; Consistency verification unit: verifies the data consistency of the same data shard and its replicas to avoid data synchronization anomalies; Authenticity verification unit: Verifies the authenticity of the data source by comparing the device identification and signature information of the data source, and prevents the injection of false data.

9. The smart park video network node data distributed storage system according to claim 1, characterized in that, The data adjustment module adopts a dynamically adjusted unit with modular deployment, which includes: Format Correction Unit: Receives abnormal feedback from the format verification unit and automatically corrects non-standard format data; Data completion unit: For missing data detected by the integrity verification unit, it performs reasonable completion or marks the missing status based on historical data patterns and related data; Quality Optimization Unit: Adjusts data resolution and encoding parameters according to application scenario requirements to improve data application quality; Redundant compression unit: The LZ4 compression algorithm is used to redundantly compress the verified valid data to reduce storage usage.

10. The smart park video network node data distributed storage system according to claim 1, characterized in that, The application interface layer provides RESTful API interfaces and SDK development toolkits, supporting integration with upper-level application systems such as park security management systems and operation and maintenance management platforms to realize data query, read, export, and delete operations.

Images