Traffic test data distributed encryption storage method, system, device and medium

Abstract

The present application relates to the technical field of data processing, and particularly provides a traffic test data distributed encryption storage method, system, device and medium, comprising: receiving and identifying the business type of traffic test data; configuring differentiated national encryption strategies for different types of data and processing into encrypted data fragments; distributing the fragments to authenticated storage nodes; dynamically adjusting the storage location of the fragments between high-speed and low-speed storage media based on a hierarchical mechanism of access frequency; recording the whole life cycle operation log, and feeding back the optimized encryption strategy and hierarchical mechanism accordingly. The present application realizes the precise matching of storage resources, security strategies and the intrinsic value of data by deeply coupling the business attributes of data with dynamic access features.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a distributed encrypted storage method, system, device, and medium for traffic test data. Background Technology

[0002] With the rapid development of autonomous driving and intelligent transportation technologies, traffic test data is characterized by its massive volume, multiple sources, sensitivity, and uneven value density. This data not only includes raw sensor streams requiring real-time processing but also encompasses high-value simulation scenarios, safety incident reports, and compliance files, exhibiting significant differences in business attributes, security levels, and access patterns. Currently, while hot-cold data tiering mechanisms based on distributed architecture and access frequency optimize storage efficiency to some extent, they suffer from a fundamental flaw: their tiering logic is completely disconnected from the inherent business types and security requirements of the data. This leads to highly sensitive scenario or event data potentially being stored on low-cost, low-performance media due to low-frequency access, compromising security and availability; while low-value historical data may consume core resources due to short-term, high-frequency access, resulting in misconfiguration and cost waste. Therefore, how to deeply integrate the business type, security level, and dynamic access characteristics of data to achieve dynamic and accurate matching of data value, storage resources, and security policies has become a core technical problem urgently needing to be solved in the field of massive traffic test data storage. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a distributed encrypted storage method, system, device and medium for traffic test data to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a distributed encrypted storage method for traffic test data, comprising: Receive traffic test data and identify the data type of the traffic test data; Configure corresponding encryption strategies for traffic test data of different data types, and process the traffic test data of the corresponding data types into encrypted data fragments according to the encryption strategies; Encrypted data is fragmented and distributed to authenticated storage nodes in a pre-built storage cluster; The level of the encrypted data fragment is determined based on the access frequency of the encrypted data fragment and the set layering mechanism, and the storage location of the data fragment between the high-speed storage medium and the low-speed storage medium is adjusted based on the level. Record the entire lifecycle operation log of the traffic test data, and optimize the encryption strategy and layering mechanism based on the entire lifecycle operation log.

[0005] In one optional implementation, traffic test data is received, and the data type of the traffic test data is identified, including: The data is identified based on its upload source or system identifier; raw streaming data from vehicle sensors or data acquisition terminals is identified as raw data type; output files from scenario building or simulation testing platforms are identified as scenario extraction type; reports from test monitoring or anomaly detection systems are identified as event analysis type; and data packets submitted to certification or regulatory agencies to meet regulatory requirements are identified as archived compliance type.

[0006] Identification is performed based on the format or naming characteristics of the data files; files that conform to the predefined scene file format or contain scene identifiers are identified as scene extraction types; files whose file names or metadata contain event numbers or exception tags are identified as event analysis types.

[0007] The metadata carried by the data is parsed and identified based on the fields that indicate business attributes; the business attribute fields include data category, testing phase or purpose description, and their values ​​directly correspond to one of the four business types.

[0008] In one optional implementation, corresponding encryption strategies are configured for traffic test data of different data types, including: For traffic test data of the original data type, the configuration uses the SM4 algorithm for encryption; For traffic test data of the scene extraction type, the configuration adopts the SM4-GCM algorithm that supports authentication encryption mode for encryption; For traffic test data of both event analysis and archived compliance types, a hybrid encryption strategy combining SM2 and SM4 algorithms is configured for encryption. For archived compliant traffic test data, the encryption strategy is further combined with blockchain notarization to achieve the immutability and existence proof of the data.

[0009] In an optional implementation, the method for constructing the storage cluster includes: Deploy one master node, multiple storage nodes, and at least two backup nodes to form a master-slave architecture; wherein, the master node is configured as a control center responsible for global scheduling, key management, and access control; the storage nodes are configured as data storage units grouped according to a predetermined strategy; the backup nodes are configured as redundant units that synchronize the master node's key management data in real time, the key management data including key information, metadata indexes, and node status information; When a storage node joins the cluster, the master node authenticates its identity based on the national cryptographic SM9 algorithm and collects its hardware fingerprint and operating status parameters for security rating. Based on the rating results, storage nodes with different security levels are assigned to different trusted storage resource pools, with the high-security-level resource pool designated for storing sensitive data. A dynamic load balancing strategy is configured in the master node. This strategy is based on an improved consistent hashing algorithm to realize dynamic mapping between data shards and storage nodes, and sets a node load threshold. When the cluster is running, the master node monitors the node load according to this strategy and triggers data shard migration when the threshold is exceeded, so as to achieve balanced utilization of storage resources.

[0010] In one optional implementation, the encrypted data fragments are distributed to authenticated storage nodes within a pre-built storage cluster, including: For each data shard, the master node selects at least two nodes belonging to different groups from the certified storage nodes as target storage nodes based on the security rating results and real-time load status of the storage nodes. The data fragments are concurrently transmitted to the selected target storage nodes for storage, and the index information of the data fragments is synchronously stored in the backup nodes. The mapping relationship between the data shards and the target storage nodes, the storage time, and the identification of the applied encryption key are updated to the distributed index library of the master node. At the same time, this distribution operation is written as a record of evidence into the blockchain traceability module.

[0011] In an optional implementation, the hierarchy of the encrypted data fragments is determined based on the access frequency of the encrypted data fragments and a set hierarchical mechanism, and the storage location of the data fragments between high-speed and low-speed storage media is adjusted based on the hierarchy, including: The master node continuously collects access logs for each data shard through the monitoring pipeline and dynamically calculates the heat score of each data shard based on a weighted decay algorithm. According to the preset heat score threshold, the data shard is automatically determined to be one of the three sub-layers: hot, warm, and cold. When a change in the heat score of a data shard causes a change in the determination result of its sub-layer, or when the continuous inaccessibility time of the current sub-layer reaches a preset migration threshold, the layering engine triggers a hierarchical migration decision; the decision includes determining the target sub-layer and the corresponding target storage medium type. For cross-sublayer migration decisions, the layering engine issues migration instructions to the data agents of the target storage node and the source storage node; the data agents, according to the instructions, migrate the data fragments from the physical location of the source storage medium to the physical location of the target storage medium and complete the update of the storage path.

[0012] In an optional implementation, the encryption strategy and layering mechanism are optimized based on the full lifecycle operation log, including: The full lifecycle operation logs are obtained from the blockchain, and by analyzing access records, decryption records, key usage records, and node performance data, the security risk patterns and periodic access patterns of the data are identified. Based on security risk patterns and periodic access patterns, corresponding optimization strategies are generated; the optimization strategies include at least one of the following: encryption strategy optimization strategy, which includes instructions to increase or decrease the encryption algorithm strength level of specified data and adjust its key rotation cycle; layered migration strategy optimization strategy, which includes instructions to perform predictive migration of specified data, or parameters to adjust the determination thresholds of hot sublayer, warm sublayer, and cold sublayer. The optimization strategy is delivered to the corresponding functional module for execution to make the optimization effective; wherein: for the encryption strategy optimization strategy, the key management module of the master node triggers the key rotation and data re-encryption process; for the layered migration strategy optimization strategy, the layered engine or the master node triggers the data migration process or updates the layered determination parameters.

[0013] Secondly, the present invention provides a distributed encrypted storage system for traffic test data, comprising: A type recognition module is used to receive traffic test data and identify the data type of the traffic test data. The encryption processing module is used to configure corresponding encryption strategies for traffic test data of different data types, and process traffic test data of corresponding data types into encrypted data fragments according to the encryption strategies. The distribution storage module is used to distribute encrypted data shards to authenticated storage nodes in a pre-built storage cluster; The data layering module is used to determine the layer of the encrypted data fragment based on the access frequency of the encrypted data fragment and the set layering mechanism, and to adjust the storage location of the data fragment between the high-speed storage medium and the low-speed storage medium based on the layer. The feedback optimization module is used to record the full lifecycle operation log of the traffic test data and optimize the encryption strategy and layering mechanism based on the full lifecycle operation log.

[0014] Thirdly, a device is provided, comprising: A distributed encrypted storage program for storing traffic test data; A processor is configured to implement the steps of the distributed encrypted storage method for traffic test data as provided in the first aspect when executing the traffic test data distributed encrypted storage program.

[0015] Fourthly, a computer-readable medium is provided, on which a traffic test data distributed encryption storage program is stored, wherein when the traffic test data distributed encryption storage program is executed by a processor, it implements the steps of the traffic test data distributed encryption storage method provided in the first aspect.

[0016] The beneficial effects of this invention are as follows: the distributed encrypted storage method, system, device, and medium for traffic test data provided by this invention achieve precise matching of storage resources, security policies, and the intrinsic value of the data by deeply coupling the business type and dynamic access frequency of the traffic test data. Its beneficial effects include: simultaneous improvement in storage performance and security, ensuring that high-value business data always receives matching encryption strength and storage performance, and significantly reducing access latency; highly intelligent resource allocation, with the system able to predict and migrate data based on access patterns, optimizing storage costs by over 40%; the system possesses self-optimization capabilities, dynamically adjusting encryption and hierarchical strategies through log feedback, enhancing overall adaptability and reliability; and, simultaneously, end-to-end blockchain traceability and fine-grained access control fully meet compliance auditing and data protection requirements, achieving secure, efficient, and compliant integrated storage of massive amounts of traffic test data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0023] The distributed encrypted storage method for traffic test data provided in this embodiment of the invention is executed by a computer device, and correspondingly, the distributed encrypted storage system for traffic test data runs on the computer device.

[0024] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be a distributed encrypted storage system for traffic test data. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0025] like Figure 1 As shown, the method includes: S1. Receive traffic test data and identify the data type of the traffic test data; S2. Configure corresponding encryption strategies for traffic test data of different data types, and process the traffic test data of the corresponding data types into encrypted data fragments according to the encryption strategies; S3. Distribute encrypted data fragments to authenticated storage nodes in the pre-built storage cluster; S4. Determine the level of the encrypted data fragment based on the access frequency of the encrypted data fragment and the set layering mechanism, and adjust the storage location of the data fragment between the high-speed storage medium and the low-speed storage medium based on the level. S5. Record the full lifecycle operation log of the traffic test data, and optimize the encryption strategy and layering mechanism based on the full lifecycle operation log.

[0026] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0027] S101. Identification based on data source path: A data source-business type mapping rule base is configured at the data access point (API gateway or dedicated receiving service). When a data stream or file begins uploading, the system first extracts its source identifier, such as the device ID of the uploading terminal, the URL path of the access service, or the digital certificate of the sending system. The rule engine then performs real-time matching based on this information. If the source identifier is registered as "Vehicle VIN_XXX Collection Terminal" or "Roadside RSU_XXX", the system will immediately mark the batch of data as the raw data type and guide it to the raw data cache queue.

[0028] If the source is identified as "Simulation Cloud Platform_Scene Generation Service" or "Scene Annotation Tool", then its output file will be marked as a scene extraction type.

[0029] If the source path points to "Test Management Platform - Anomaly Reporting Interface" or the log aggregation system, then the report will be identified as an event analysis type.

[0030] For data packets that are actively uploaded through the "Compliance Reporting Gateway" and whose target address is a preset regulatory agency receiving end, an archived compliance type label is automatically assigned to them.

[0031] S102. Identification based on file characteristics and naming conventions: For data that cannot be accurately identified based on its source alone, or as a form of double verification, the system initiates the file feature analysis module in parallel.

[0032] Format parsing: The system maintains a predefined list of scene file formats (such as .scenario.xml, .scene.json). When a file is entered, its extension and file header magic number are quickly checked. If they match, it is classified as a scene extraction type.

[0033] Rule matching: The system has a built-in named regular expression rule library. For example, for files that match the pattern Event_[0-9]{8}_[AZ]{2}.zip, they are automatically identified as event packages containing event numbers and classified as event analysis types; folders or file packages containing the keyword Scenario_Iteration_v* are classified as scene extraction types.

[0034] S103. Identification based on embedded metadata: To achieve the most accurate classification, the system requires that standardized business metadata be included or automatically extracted in the data upload protocol. After data access, the metadata parser extracts key fields from it: data_category: Its value is an enumeration, such as RAW_STREAM (raw stream), TEST_SCENARIO (test scenario), SAFETY_INCIDENT (security event), CERTIFICATION_PACKAGE (certification package). This field directly maps to four business types.

[0035] test_phase: For example, "Data Collection", "Simulation & Validation", "Incident Analysis", and "Regulatory Submission". The system determines the type based on a preset mapping table.

[0036] intended_use: such as "Model Training", "Regression Testing", "Root Cause Analysis", "Audit Trail", to further assist in confirming the business type.

[0037] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0038] S201. Matching and Configuration of Encryption Policies: Policy rule base: The encryption policy engine has a built-in policy rule base, whose core rules are: Rule 1: If the data's business type label is "raw data type", then configure its encryption algorithm as SM4-CTR mode with a key length of 128 bits. CTR mode is chosen to meet the requirements of high throughput and parallel encryption of the raw data stream.

[0039] Rule 2: If the business type label is "Scenario Extraction Type", then configure its encryption algorithm as SM4-GCM mode with a key length of 128 bits. Using GCM mode ensures confidentiality while providing integrity and authentication, guaranteeing that the simulation test scenario cannot be tampered with.

[0040] Rule 3: If the business type label is "Event Analysis Type" or "Archived Compliance Type," then configure a hybrid encryption strategy for it. Specifically: use the SM2 algorithm (256-bit key) to encrypt a random data encryption key, and then use this data encryption key to encrypt the actual data using the SM4 algorithm. This strategy combines the asymmetric security features of SM2 with the symmetric encryption efficiency of SM4.

[0041] Special Enhancement Processing: For "Archived Compliance Type" data, after applying the above hybrid encryption strategy, the system will automatically trigger the blockchain notarization process. After encryption, the hash value (such as SM3 hash), encryption timestamp, key identifier, and other digest information of the data packet will be submitted to the blockchain network to generate an immutable notarization record.

[0042] S202. Key generation, distribution, and acquisition: The key management service generates or distributes keys for the national cryptographic algorithm according to the instructions of the encryption policy engine.

[0043] For SM4 encryption (rules one and two), the key management service obtains a currently active key from the dynamic key pool of the corresponding business type.

[0044] For the hybrid encryption strategy (Rule 3), the key management service performs two steps: first, it generates a one-time SM4 data encryption key; then, it calls the SM2 algorithm and uses a pre-configured SM2 public key that matches the data security level to encrypt the SM4 key, forming the "key ciphertext".

[0045] Finally, the symmetric key (or SM4 key ciphertext) used to encrypt the data is securely transmitted to the encryption execution module.

[0046] S203. Data Encryption and Fragmentation Processing Execution: Encryption Execution: The data fragmentation processor receives the data blocks to be processed, along with the corresponding encryption instructions and key materials. It invokes the integrated national cryptographic algorithm hardware acceleration card or software library to strictly complete the encryption operation according to the configuration. For hybrid encryption strategies, the processor first uses the acquired SM4 key to encrypt the data body and binds the SM4 key ciphertext as part of the metadata to the encrypted data.

[0047] Adaptive Fragmentation: After encryption, the processor divides the encrypted data into multiple fragments based on the data's characteristics (such as whether it is a large unstructured video file) and a preset adaptive fragmentation algorithm. The fragment size is dynamically determined; for example, structured data fragments are 1-2GB, and unstructured video data fragments are 5-10GB. Each fragment generates an independent CRC-32 checksum.

[0048] Fragmentation and encapsulation: Each encrypted data fragment and its metadata (including the original data ID, fragment sequence number, encryption algorithm identifier, key identifier or key ciphertext, and checksum) are encapsulated into a complete storage unit, awaiting distribution.

[0049] S204. Implementation of Blockchain Notarization for Archived Data: For archived compliant data, after encryption and sharding are completed, the system packages its overall hash value, key metadata (such as certification report number and submission time), and SM2 public key fingerprint, and sends it to the permissioned blockchain network composed of master nodes, backup nodes, and regulatory nodes.

[0050] Blockchain nodes upload this evidence to the chain via a consensus mechanism, returning a unique transaction hash and block height. This credential will be stored in association with archived data, serving as the ultimate basis for verifying data integrity and existence during future audits.

[0051] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0052] Building a storage cluster is a layered and orderly process that covers three major stages: hardware deployment, security access control, and operational policy configuration. The goal is to establish a secure, reliable, and elastically scalable data storage infrastructure.

[0053] 1. Hardware deployment and infrastructure setup: Node roles and specifications: Master node: Deployed using a high-performance server, configured with dual Intel Xeon Platinum series CPUs, 128GB or more of ECC memory, and a high-speed NVMe SSD system disk. Core service software such as cluster management, key management, scheduling engine, and blockchain client are installed.

[0054] Storage nodes: High-density storage servers are used, configured with large-capacity hard disk arrays (such as 40TB HDDs) and high-speed cache disks (such as 2TB SSDs). Physical or logical groups are formed according to the data center rack location or the planned data type (such as "raw data - North China", "scenario data - East China"). Each group is recommended to have 3-5 nodes to achieve redundancy within the group.

[0055] Backup nodes: Configure at least two servers with the same hardware specifications as the primary node, deployed in remote data centers at least 50 kilometers away from the primary node, and interconnected via dedicated lines. These servers serve as hot standby instances running all critical management services of the primary node.

[0056] Network interconnection: All nodes are interconnected via a high-speed fiber optic network (core switch ≥100Gbps), and industrial-grade 5G routers are deployed as backup communication links. Inter-node communication is encrypted using an IPsec-based VPN tunnel.

[0057] 2. Node security access control and trusted resource pool partitioning: SM9 authentication implementation: During cluster initialization or when a new node joins, the master node's security access service initiates the authentication process. Storage nodes must provide their pre-issued SM9 digital certificates. The master node verifies the certificate's validity and performs two-way authentication using the SM9 algorithm.

[0058] Hardware fingerprinting and security rating: After authentication, the master node collects the node's hardware fingerprint (such as the endorsement key, CPU ID, and MAC address hash value from the TPM module) and initial operating status parameters (such as firmware version and open port list). This information is input into the security rating model, which scores the node according to preset rules (such as: TPM 2.0 is available, the firmware is the latest security version, and no high-risk ports are open), classifying the node into three security levels: A (high), B (medium), and C (low).

[0059] Trusted resource pooling management: Based on the rating results, the master node assigns nodes to different trusted storage resource pools. For example, all Level A nodes are included in the "Core Sensitive Data Pool," specifically for storing data such as event analysis and archiving compliance; Level B and Level C nodes are included in the "General Data Pool" and "Temporary / Cache Data Pool," respectively. This mapping relationship is recorded in the node status database for subsequent querying by the data distribution scheduler.

[0060] 3. Configuration and operation of dynamic load balancing strategies: In the master node's scheduling engine, a virtual hash ring is constructed. Each storage node is mapped to multiple virtual points on the ring based on its unique identifier (such as the hash value of security level + data center number + device ID). Unlike classic algorithms, the improvement of this method lies in that the "weight" of each virtual point is dynamically associated with the node's real-time remaining storage capacity and CPU utilization, making new data shards more likely to be written to node regions with lower loads.

[0061] Threshold monitoring and migration triggering: The scheduling engine has a built-in monitoring agent that continuously collects performance metrics from all storage nodes (default sampling interval 5 seconds). A comprehensive load threshold is set for each node (default 80%, adjustable). When the load on a node consistently exceeds the threshold (e.g., for 3 consecutive periods), the scheduling engine identifies it as a "hotspot node."

[0062] Automated data sharding migration: For "hot nodes," the scheduling engine calculates new target nodes for shards of the data stored on them based on the consistent hash ring and node resource pool attributes (prioritizing the node with the lowest load within the same resource pool). Subsequently, data migration instructions are sent to both the source and target nodes. The migration process is executed asynchronously in the background at the "shard" granularity, and data consistency is ensured through a verification mechanism. During this process, data read requests are automatically redirected, achieving a seamless migration.

[0063] The distribution of encrypted data shards is an automated, strongly consistent process driven by the master node's distribution scheduler core. This process ensures that each data shard is securely, reliably, and efficiently placed within the storage cluster. The specific steps are as follows: S301. Intelligent selection of target storage nodes: Selection strategy execution: After the data sharding processor completes the encryption and encapsulation of a data shard, it submits a distribution request to the master node's distribution scheduler. The distribution scheduler then invokes the node selection strategy engine.

[0064] Multi-factor decision-making: The engine makes decisions based on two core factors: Security and compliance: The security level of the business type to which the data shards in the query request belong (such as originating from the event analysis layer) determines whether the candidate nodes must come from a matching high-security-level resource pool (Level A node pool).

[0065] Load and Topology Optimization: Obtain the real-time load status (CPU, memory, storage I / O utilization) of each node from the eligible node pool. Based on an improved consistent hashing algorithm and a "cross-group placement" strategy, calculate 2 to 3 (configurable) target nodes. These nodes must not only have low current load but also belong to different physical or logical groups (e.g., located in different racks or different data center availability zones) to mitigate single point of failure risk.

[0066] Finally, a distribution instruction containing a list of target nodes is generated.

[0067] S302. Concurrent transmission and storage of fragmented data: Concurrent transmission initiation: The distribution scheduler fragments encrypted data and sends distribution instructions concurrently to all selected target storage nodes via a high-speed data channel (such as RDMA over Converged Ethernet). This concurrent transmission mechanism maximizes network bandwidth utilization and significantly reduces distribution latency.

[0068] Node Storage and Verification: After receiving the data fragment, the data broker service on each target storage node first verifies its CRC-32 checksum to ensure transmission integrity. Upon successful verification, based on the service type and access frequency indications in the fragment metadata, it writes the fragment to the corresponding local storage medium (such as an SSD cache or HDD storage area) and returns a confirmation signal indicating successful storage.

[0069] Index synchronization to backup nodes: Simultaneously, the distribution scheduler sends the simplified index information of the data shard (including unique shard ID, hash value, data size, and the main business layer to which it belongs) to all backup nodes. Backup nodes store this information in their local high-availability index database, ensuring that even if the primary node is temporarily unavailable, the system can still locate the data through the backup nodes.

[0070] S303. Global Metadata Update and Blockchain Evidence Storage: Update the distributed index: After receiving successful confirmation from all target storage nodes, the distribution scheduler performs an eventual consistency commit. It writes a complete mapping record to the master node's distributed index, including: data shard ID, a list of target node physical addresses for each replica, storage timestamps, the encryption algorithm identifier used (e.g., SM4-GCM), and the corresponding unique key identifier. This index supports millisecond-level multidimensional queries.

[0071] Blockchain Operation Traceability: Next, the distribution scheduler encapsulates the key information of this distribution operation (including: shard ID, hash value, target node list, distribution time, operator / system identifier) ​​into a notarized transaction and submits it to the blockchain traceability module. This module consists of blockchain clients deployed on master and backup nodes. It packages the transaction into the chain using a consensus algorithm (such as RAFT), generating a time-series, immutable distribution notarization record. The block hash value of this record will be associated with the metadata of that data shard for future auditing and traceability.

[0072] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0073] S401. Access Popularity Calculation and Dynamic Layering Determination: Monitoring Pipeline and Log Collection: Deploy lightweight log collectors in the data access gateway and the data broker on each storage node. All access requests (read and write) to data shards are captured, generating lightweight access logs containing only the data shard ID, access timestamp, operation type, and user ID. These logs are pushed in real-time to the log aggregation module of the tiered engine via a message queue (such as Kafka).

[0074] Weighted decay popularity algorithm implementation: The hierarchical engine maintains a dynamic popularity score for each active data shard. The engine runs periodically (e.g., hourly) to process access logs from a past calculation window (e.g., 24 hours). The core formula of the algorithm is: New popularity score = Original popularity score * Decay factor + Σ (Access weight within the current window). Where: A decay factor (such as 0.95) ensures that the impact of historical visits decays exponentially over time, making the score more reflective of recent popularity.

[0075] Access weights are dynamically adjusted based on the type of operation. For example, a read operation has a weight of 1.0, while a write operation (often accompanied by subsequent reads) has a weight of 1.5.

[0076] Automatic sub-layer labeling: The engine compares the calculated latest popularity score with the preset stratification threshold and automatically labels the data shards with sub-layers. Heat score > threshold H (e.g., 10 points): marked as "thermal sublayer".

[0077] Threshold L (e.g., 2 points) < heat score ≤ threshold H: marked as "warm layer".

[0078] Heat score ≤ threshold L: marked as "cold sublayer".

[0079] Initially, new data is placed into the "hot sub-layer" by default. Layer labels are recorded in the enhanced metadata database.

[0080] S402. Hierarchical migration decision triggering mechanism: Condition monitoring: The layered engine continuously monitors state changes in two aspects to trigger migration decisions: Score-driven change: When the latest popularity score of a data shard causes a change in the label of its sub-layers (e.g., from "hot" to "warm"), a migration assessment is immediately triggered.

[0081] Time-driven change: For each sub-layer, the system sets a continuous no-access migration threshold (e.g., 7 days for hot sub-layers, 30 days for warm sub-layers). The engine checks the "last access time" of each data shard. If it exceeds the threshold corresponding to its current sub-layer, a migration assessment is triggered regardless of the heat score (usually migrating to a colder layer).

[0082] Intelligent decision-making: After the evaluation is triggered, the decision-making module of the hierarchical engine will execute: Target sub-layer determination: Based on the triggering reason, determine the target sub-layer (e.g., downgrade if the score drops, downgrade if the time expires, upgrade if the access event "warms up").

[0083] Cost-benefit analysis: For decisions regarding migration to a cold tier, the engine estimates the cost of the migration operation (computation and network bandwidth consumption) and the storage cost savings after the migration. If the cost savings significantly outweigh the operational costs, the migration is approved; otherwise, the migration is postponed.

[0084] Target storage medium specification: Based on the target sub-layer, determine the corresponding storage medium type from the policy library (e.g., "hot sub-layer" corresponds to SSD, "warm sub-layer" corresponds to HDD, and "cold sub-layer" corresponds to archive storage or tape).

[0085] S403. Execution of cross-sublayer data migration: Migration Instruction Generation and Issuance: The tiered engine generates specific data migration instructions, including: data shard ID, source storage node and path, target storage node and path, target storage media type, and operation priority (usually a low-priority background operation). These instructions are then issued to the data agents on the source and target nodes via control channels.

[0086] Data agent performs migration: After receiving the instruction, the source data agent reads data fragments from the specified path and performs integrity verification before transmission.

[0087] The target data agent on the target node prepares the storage location according to the target media type (such as SSD or HDD mount point) required by the instruction, and receives the data stream from the source node.

[0088] The migration process uses asynchronous background replication. To ensure business continuity, during the migration, access requests to the data shards being migrated are still handled by the source node until the migration is complete.

[0089] Metadata and path updates: After the migration is complete, the target data broker sends a success confirmation to the tiered engine. The engine then performs atomic update operations: In the enhanced metadata database, the physical storage path of the data shard is updated to point to the new location.

[0090] In the distributed index, update the location mapping of the data shard.

[0091] Notify the master node scheduler to update its routing information.

[0092] Once the source node data is confirmed to be no longer referenced, the data broker asynchronously cleans up the old data copies to free up space.

[0093] The entire migration process is transparent to upper-layer applications, enabling dynamic and optimized configuration of storage resources.

[0094] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0095] S501. Aggregation and in-depth analysis of full lifecycle logs: Log Sources and Data Aggregation: The engine continuously aggregates logs throughout their entire lifecycle from multiple sources. Blockchain traceability module: Provides tamper-proof data access, modification, and operation audit records.

[0096] Key management service: Provides records of key generation, use, rotation, and successful / failed decryption.

[0097] Monitoring pipeline: Provides fine-grained data access modes (such as sequential read, random read, user ID) and real-time and historical performance metrics (IOPS, latency, CPU / memory utilization) for each storage node.

[0098] Pattern recognition algorithm execution: The engine uses time series analysis, association rule mining, and other algorithms to process the aggregated logs, and the core identification is two types of patterns: Security risk patterns: For example, unusually high frequency of decryption failures for the same data shard, access clusters from uncommon geographical locations, or keys being accessed at unexpected times may indicate brute-force attempts or internal leakage risks.

[0099] Business access patterns: For example, spectrum analysis reveals that certain test scenario data is accessed intensively every Monday at 9:00 AM, or certain archived data is read in batches at the end of each quarter to generate reports. The engine can also identify special access patterns of "high traffic, low frequency".

[0100] S502. Generation and decision-making of optimization strategies: Strategy Reasoning and Generation: Based on the identified patterns, the engine reasones according to a pre-built optimization rule base to generate executable optimization strategy instructions or parameter adjustment suggestions. Encryption strategy optimization strategy: If the actual access value and frequency of specific data (such as a high-definition map of a certain road segment) are found to far exceed its initial "generally sensitive" setting, then the following instruction is generated: {Target data ID, Operation: Increase encryption level, New algorithm: SM2+SM4, New key rotation cycle: 24 hours}.

[0101] If a stable quarter-end access pattern is found for a certain type of "cold data", then generate the instruction: {Target data type, Operation: Adjust key period, New period: Align with the quarterly report after generation}.

[0102] Layered migration strategy optimization strategy: For the identified periodic access patterns (such as scene calls on Monday mornings), generate predictive migration instructions: {target data ID set, operation: pre-warm migration, target sub-layer: hot, execution time: Sunday 22:00}.

[0103] Based on the identified "high traffic, low frequency" access characteristics, threshold adjustment parameters are generated: {For data with a file size > 1TB, the hot sublayer judgment threshold is dynamically adjusted: access frequency ≥ 1 time / day}.

[0104] Strategy Risk Assessment and Approval: For high-impact strategies involving security level upgrades or large-scale data migration, the engine can submit them to the management interface for administrator review and confirmation; for regular optimization strategies, they can be set to be automatically approved and executed.

[0105] S503. Automated execution and activation of optimization strategies: Policy distribution and routing: The generated optimization policies are encapsulated as standard messages and distributed via the internal event bus. The message header contains a policy type identifier, which the system uses to route the policy to the corresponding execution module.

[0106] Encryption policy execution: When the key management module receives the "Encryption policy optimization policy" instruction, if it involves an encryption algorithm upgrade (such as SM4->SM2+SM4), the following process will be automatically triggered: The target data is asynchronously re-encrypted using a new high-strength algorithm and key.

[0107] After re-encryption is complete, update the encryption algorithm identifier and key identifier in the metadata, and replace the old ciphertext with the new key ciphertext.

[0108] The key rotation schedule for this data will be updated according to the period set by the new instructions.

[0109] Execution of the tiered migration strategy: When the tiered engine or master node scheduler receives the "tiered migration strategy optimization strategy" instruction: For predictive migration instructions, the hierarchical engine incorporates them into the scheduling plan, triggering a standard data migration workflow at a specified time to move the data to high-speed storage media in advance.

[0110] When the threshold adjustment parameter is applied, the hierarchical engine immediately updates its internal decision model parameters. Subsequently, all data popularity score evaluations and sub-layer labeling will be based on the new threshold, thus systematically influencing subsequent migration decisions.

[0111] Execution status monitoring and feedback: Each execution module feeds back the results (success, failure, progress) of strategy execution to the log analysis engine. The engine records this feedback to evaluate the optimization effect and iteratively improve its analysis model, thus forming a continuously learning and self-evolving intelligent closed loop.

[0112] In some embodiments, the distributed encrypted storage system for traffic test data may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the distributed encrypted storage system for traffic test data may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality of distributed encrypted storage of traffic test data.

[0113] In this embodiment, the traffic test data distributed encrypted storage system can be divided into multiple functional modules according to its functions, such as... Figure 2 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0114] A type recognition module is used to receive traffic test data and identify the data type of the traffic test data. The encryption processing module is used to configure corresponding encryption strategies for traffic test data of different data types, and process traffic test data of corresponding data types into encrypted data fragments according to the encryption strategies. The distribution storage module is used to distribute encrypted data shards to authenticated storage nodes in a pre-built storage cluster; The data layering module is used to determine the layer of the encrypted data fragment based on the access frequency of the encrypted data fragment and the set layering mechanism, and to adjust the storage location of the data fragment between the high-speed storage medium and the low-speed storage medium based on the layer. The feedback optimization module is used to record the full lifecycle operation log of the traffic test data and optimize the encryption strategy and layering mechanism based on the full lifecycle operation log.

[0115] Figure 3 The distributed encrypted storage method for traffic test data provided in this application embodiment can be applied to devices. Those skilled in the art will understand that the device structure involved in the embodiments of this invention does not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0116] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0117] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.

[0118] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0119] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0120] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0121] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0122] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0123] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0124] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0126] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A distributed encrypted storage method for traffic test data, characterized in that, include: Receive traffic test data and identify the data type of the traffic test data; Configure corresponding encryption strategies for traffic test data of different data types, and process the traffic test data of the corresponding data types into encrypted data fragments according to the encryption strategies; Encrypted data is fragmented and distributed to authenticated storage nodes in a pre-built storage cluster; The level of the encrypted data fragment is determined based on the access frequency of the encrypted data fragment and the set layering mechanism, and the storage location of the data fragment between the high-speed storage medium and the low-speed storage medium is adjusted based on the level. Record the entire lifecycle operation log of the traffic test data, and optimize the encryption strategy and layering mechanism based on the entire lifecycle operation log.

2. The method according to claim 1, characterized in that, Receive traffic test data and identify the data type of the traffic test data, including: The data is identified based on its upload source or system identifier; raw streaming data from vehicle sensors or data acquisition terminals is identified as raw data type; output files from scenario building or simulation testing platforms are identified as scenario extraction type; reports from test monitoring or anomaly detection systems are identified as event analysis type; and data packets submitted to certification or regulatory agencies to meet regulatory requirements are identified as archived compliance type. Identification is performed based on the format or naming characteristics of the data files; files that conform to the predefined scene file format or contain scene identifiers are identified as scene extraction types; files whose file names or metadata contain event numbers or exception tags are identified as event analysis types. The metadata carried by the data is parsed and identified based on the fields that indicate business attributes; the business attribute fields include data category, testing phase or purpose description, and their values ​​directly correspond to one of the four business types.

3. The method according to claim 1, characterized in that, Configure corresponding encryption strategies for traffic test data of different data types, including: For traffic test data of the original data type, the configuration uses the SM4 algorithm for encryption; For traffic test data of the scene extraction type, the configuration adopts the SM4-GCM algorithm that supports authentication encryption mode for encryption; For traffic test data of both event analysis and archived compliance types, a hybrid encryption strategy combining SM2 and SM4 algorithms is configured for encryption. For archived compliant traffic test data, the encryption strategy is further combined with blockchain notarization to achieve the immutability and existence proof of the data.

4. The method according to claim 1, characterized in that, The method for constructing the storage cluster includes: Deploy one master node, multiple storage nodes, and at least two backup nodes to form a master-slave architecture; wherein, the master node is configured as a control center responsible for global scheduling, key management, and access control; the storage nodes are configured as data storage units grouped according to a predetermined strategy; the backup nodes are configured as redundant units that synchronize the master node's key management data in real time, the key management data including key information, metadata indexes, and node status information; When a storage node joins the cluster, the master node authenticates its identity based on the national cryptographic SM9 algorithm and collects its hardware fingerprint and operating status parameters for security rating. Based on the rating results, storage nodes with different security levels are assigned to different trusted storage resource pools, with the high-security-level resource pool designated for storing sensitive data. A dynamic load balancing strategy is configured in the master node. This strategy is based on an improved consistent hashing algorithm to realize dynamic mapping between data shards and storage nodes, and sets a node load threshold. When the cluster is running, the master node monitors the node load according to this strategy and triggers data shard migration when the threshold is exceeded, so as to achieve balanced utilization of storage resources.

5. The method according to claim 4, characterized in that, Distribute encrypted data shards to authenticated storage nodes in a pre-built storage cluster, including: For each data shard, the master node selects at least two nodes belonging to different groups from the certified storage nodes as target storage nodes based on the security rating results and real-time load status of the storage nodes. The data fragments are concurrently transmitted to the selected target storage nodes for storage, and the index information of the data fragments is synchronously stored in the backup nodes. The mapping relationship between the data shards and the target storage nodes, the storage time, and the identification of the applied encryption key are updated to the distributed index library of the master node. At the same time, this distribution operation is written as a record of evidence into the blockchain traceability module.

6. The method according to claim 5, characterized in that, The hierarchy of the encrypted data fragments is determined based on the access frequency of the encrypted data fragments and the established hierarchical mechanism, and the storage location of the data fragments between high-speed and low-speed storage media is adjusted based on the hierarchy, including: The master node continuously collects access logs for each data shard through the monitoring pipeline and dynamically calculates the heat score of each data shard based on a weighted decay algorithm. According to the preset heat score threshold, the data shard is automatically determined to be one of the three sub-layers: hot, warm, and cold. When a change in the heat score of a data shard causes a change in the determination result of its sub-layer, or when the continuous inaccessibility time of the current sub-layer reaches a preset migration threshold, the layering engine triggers a hierarchical migration decision; the decision includes determining the target sub-layer and the corresponding target storage medium type. For cross-sublayer migration decisions, the layering engine issues migration instructions to the data agents of the target storage node and the source storage node; the data agents, according to the instructions, migrate the data fragments from the physical location of the source storage medium to the physical location of the target storage medium and complete the update of the storage path.

7. The method according to claim 1, characterized in that, The encryption strategy and layering mechanism are optimized based on the full lifecycle operation logs, including: The full lifecycle operation logs are obtained from the blockchain, and by analyzing access records, decryption records, key usage records, and node performance data, the security risk patterns and periodic access patterns of the data are identified. Based on security risk patterns and periodic access patterns, corresponding optimization strategies are generated; the optimization strategies include at least one of the following: encryption strategy optimization strategy, which includes instructions to increase or decrease the encryption algorithm strength level of specified data and adjust its key rotation cycle; layered migration strategy optimization strategy, which includes instructions to perform predictive migration of specified data, or parameters to adjust the determination thresholds of hot sublayer, warm sublayer, and cold sublayer. The optimization strategy is delivered to the corresponding functional module for execution to make the optimization effective; wherein: for the encryption strategy optimization strategy, the key management module of the master node triggers the key rotation and data re-encryption process; for the layered migration strategy optimization strategy, the layered engine or the master node triggers the data migration process or updates the layered determination parameters.

8. A distributed encrypted storage system for traffic test data, characterized in that, include: A type recognition module is used to receive traffic test data and identify the data type of the traffic test data. The encryption processing module is used to configure corresponding encryption strategies for traffic test data of different data types, and process traffic test data of corresponding data types into encrypted data fragments according to the encryption strategies. The distribution storage module is used to distribute encrypted data shards to authenticated storage nodes in a pre-built storage cluster; The data layering module is used to determine the layer of the encrypted data fragment based on the access frequency of the encrypted data fragment and the set layering mechanism, and to adjust the storage location of the data fragment between the high-speed storage medium and the low-speed storage medium based on the layer. The feedback optimization module is used to record the full lifecycle operation log of the traffic test data and optimize the encryption strategy and layering mechanism based on the full lifecycle operation log.

9. A distributed encrypted storage device for traffic test data, characterized in that, include: A distributed encrypted storage program for storing traffic test data; A processor, configured to implement the steps of the distributed encrypted storage method for traffic test data as described in any one of claims 1-7 when executing the distributed encrypted storage program for traffic test data.

10. A computer-readable medium storing a computer program, characterized in that, The readable medium stores a traffic test data distributed encryption storage program, which, when executed by a processor, implements the steps of the traffic test data distributed encryption storage method as described in any one of claims 1-7.

Images