Multi-source motion data management system and method under scalable distributed storage architecture

Abstract

The present application discloses a multi-source action data management system and method under an extensible distributed storage architecture, which relates to the field of data management technology. The method includes accessing the original collected data information collected by the data acquisition module based on a multi-source data interface and outputting a data access completion signal; compressing the original collected data information to obtain multi-source compressed data information, calibrating the timestamp of the multi-source compressed data information, and marking the collection parameters of each data in the multi-source compressed data information; obtaining data dynamic sharding strategy information according to the data type and collection time of the multi-source compressed data information, judging the health of each storage node in the storage cluster, screening effective storage nodes, and distributing each data to the effective storage nodes based on the data dynamic sharding strategy information, judging whether the storage cluster needs to be expanded, and if expansion is required, expanding based on the data migration algorithm. The present application has the effect of improving the efficiency of multi-source action data management.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a multi-source action data management system and method under an extensible distributed storage architecture. Background Art

[0002] With the continuous advancement of informatization, data has become one of the most important assets in modern society. The diversification of data sources and the explosive growth in data volume make data management particularly important. Multi-source motion data management is a key technology for building a comprehensive data view. It facilitates unified management, analysis, and application in large-scale motion capture scenarios. Therefore, multi-source motion data management within a scalable distributed storage architecture is particularly important.

[0003] Existing multi-source motion data management methods refer to data collection and integration, data preprocessing, storage solution design, metadata management, data storage implementation, and indexing and retrieval optimization. However, in the existing multi-source motion data management methods, the data sources are extremely diverse during the motion data collection process, the data storage formats are different, and the data dimensions vary greatly. As a result, during the data processing process, data of different modalities are difficult to accurately match and fuse. In actual motion capture scenarios, the peak value of simultaneous writing of multiple cameras can reach 10Gbps. Traditional distributed systems cannot withstand such high writing pressure and are prone to data congestion and loss, resulting in low efficiency of multi-source motion data management. There is room for improvement. Summary of the Invention

[0004] In order to improve the efficiency of multi-source motion data management, the present application provides a multi-source motion data management system and method under a scalable distributed storage architecture.

[0005] In the first aspect, the multi-source action data management method under the scalable distributed storage architecture provided by this application adopts the following technical solutions:

[0006] A multi-source action data management method under a scalable distributed storage architecture includes:

[0007] Obtain a multi-source data interface, perform access operations on the original data information collected by the data acquisition module based on the multi-source data interface, and output a data access completion signal after data access is completed;

[0008] After receiving the data access completion signal, the original collected data information is compressed to obtain multi-source compressed data information, the multi-source compressed data information is time-stamped, and the collection parameters of each data in the multi-source compressed data information are marked;

[0009] The data dynamic sharding strategy information is obtained based on the data type and collection time of each data in the multi-source compressed data information. The storage cluster is obtained and the health of each storage node in the storage cluster is determined. The valid storage nodes are selected and the data is dispersed and stored to the valid storage nodes based on the data dynamic sharding strategy information. The storage cluster is determined to determine whether expansion is required. If expansion is required, the expansion operation is performed based on the data migration algorithm.

[0010] Obtain and perform query and read operations on data in the storage cluster based on the user service layer;

[0011] Construct an R* tree index structure based on each data in the storage node and create a metadata map structure, wherein the R* tree index structure and the metadata map structure are combined to form a data location query structure;

[0012] Upload the data in the storage node to the Merkle tree of the blockchain to determine whether the data in the storage node has been modified. If modified, generate an incremental snapshot and add it to the Merkle tree. Rollback operations can be performed based on the data collection time information;

[0013] The data security of each data in the storage node is protected based on dynamic desensitization technology and the national encryption SM4 algorithm.

[0014] Preferably, a multi-source data interface and a data acquisition module are obtained, and a signal connection link is established between the multi-source data interface and the data acquisition module, wherein the multi-source data interface is a unified access interface supporting multiple protocols;

[0015] Acquire original collected data information based on the data acquisition module, wherein the original collected data information includes optical data information, IMU data information, and tactile data information;

[0016] The original collected data information is accessed based on the multi-source data interface, and a data access completion signal is output after the access is completed. The multi-source data interface adopts the NVMe protocol when accessing optical data information, the multi-source data interface adopts the MQTT message queue when accessing IMU data information, and the multi-source data interface adopts the gRPC stream when accessing tactile data information.

[0017] Preferably, after receiving the data access completion signal, the optical data information is compressed to obtain optical compressed data information, the IMU data information is compressed to obtain IMU compressed data information, and the tactile data information is compressed to obtain tactile compressed data information, and the optical compressed data information, the IMU compressed data information, and the tactile compressed data information are combined to form multi-source compressed data information;

[0018] Obtaining the acquisition time of each data in the optical compression data information, namely, the optical data acquisition timestamp; obtaining the acquisition time of each data in the IMU compression data information, namely, the IMU data acquisition timestamp; obtaining the acquisition time of each data in the tactile compression data information, namely, the tactile data acquisition timestamp;

[0019] The optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp are combined to form data acquisition time information;

[0020] Performing time stamp calibration on the optical compressed data information, the IMU compressed data information, and the tactile data compressed data information according to the optical data acquisition time stamp, the IMU data acquisition time stamp, and the tactile data acquisition time stamp, and outputting a time stamp calibration completion signal after calibration;

[0021] After receiving the timestamp calibration completion signal, the data type information, device ID information, and spatial coordinate information of each data in the optical compression data information, IMU compression data information, and tactile compression data information are judged and marked to obtain associated parameter marking information.

[0022] Preferably, the multi-source compressed data information is preliminarily divided into a plurality of sub-shard data according to the data type information to obtain preliminary data fragment information;

[0023] According to the data collection time information, the data initial sharding information is further divided into multiple sub-sharding data to obtain data dynamic sharding strategy information;

[0024] Acquire a storage cluster, the storage cluster including multiple storage nodes, detect the health of each storage node, and if the health of a storage node is low, mark the storage node as an invalid storage node, and mark all other storage nodes except the invalid storage node as valid storage nodes;

[0025] Based on the data dynamic sharding strategy information, each sub-shard data is dispersed and stored on each valid storage node;

[0026] Based on the currently known storage capacity and the total storage capacity of the storage cluster, determine whether the storage cluster needs to add new storage nodes for expansion. If expansion is required, perform dynamic expansion based on the data migration algorithm of the consistent hash ring;

[0027] According to the access frequency and timeliness of each data in the multi-source compressed data information, each data in the multi-source compressed data information is divided into cold and hot data layers to obtain a cold and hot data division result. The cold and hot data division result includes cold data and hot data, and the cold data is further compressed based on the erasure code technology.

[0028] Preferably, a user service layer is obtained, wherein the user service layer includes a query engine and a machine learning interface;

[0029] A signal connection link is established between the query engine and the storage cluster. Based on the query engine, a user inputs a query instruction, and the query engine filters and displays data in the storage cluster that meets the query instruction.

[0030] A signal connection link is created between the machine learning interface and the storage cluster, and data in the storage cluster is read based on the machine learning interface.

[0031] Preferably, an R* tree index structure is constructed for the data collection time information of each data in the storage node and the spatial coordinate information in the associated parameter tag information;

[0032] Creating a metadata graph structure based on a Neo4j graph database. The metadata graph structure includes multiple nodes and edges. The nodes include device ID nodes, data type nodes, and collection time nodes. The edges represent the relationship between the data in the storage nodes.

[0033] The R*-tree index structure and the metadata graph structure are combined to form a data location query structure.

[0034] Preferably, the data in each storage node of the storage cluster is uploaded to the Merkle tree of the blockchain, wherein the leaf nodes of the Merkle tree store the hash value of each data, and the non-leaf nodes store the hash value of their child nodes;

[0035] Determine whether the data in each storage node of the storage cluster has been modified. If so, generate an incremental snapshot after the modification and add the incremental snapshot to the Merkle tree.

[0036] If the user encounters data errors or needs to restore historical data, the data can be rolled back based on the data collection time information.

[0037] Preferably, the data in the storage node are processed in real time according to the dynamic desensitization technology, and irreversible noise is added to the data in the storage node to meet the K-anonymity ≥ 5;

[0038] Obtain a hardware security module and encrypt and store cold data based on the national secret SM4 algorithm, where the key is managed by the hardware security module.

[0039] Secondly, this application provides a multi-source action data management system under a scalable distributed storage architecture, which adopts the following technical solutions:

[0040] A multi-source motion data management system under a scalable distributed storage architecture, including:

[0041] A data access module is configured to obtain a multi-source data interface, access the original data information collected by the data acquisition module based on the multi-source data interface, and output a data access completion signal after the data access is completed;

[0042] The data preprocessing module is configured to, upon receiving a data access completion signal, compress the original collected data information to obtain multi-source compressed data information, perform time stamp calibration on the multi-source compressed data information, and mark the collection parameters of each data in the multi-source compressed data information;

[0043] The storage design module is configured to obtain data dynamic sharding strategy information based on the data type and collection time of each data in the multi-source compressed data information, obtain the storage cluster and determine the health of each storage node in the storage cluster, select valid storage nodes, and disperse the data to the valid storage nodes based on the data dynamic sharding strategy information. It determines whether the storage cluster needs to be expanded. If expansion is required, expansion is performed based on the data migration algorithm.

[0044] The service layer acquisition module is configured to acquire and perform query operations and read operations on various data in the storage cluster based on the user service layer;

[0045] A location query structure creation module is configured to construct an R* tree index structure based on each data in the storage node and create a metadata map structure, wherein the R* tree index structure and the metadata map structure are combined to form a data location query structure;

[0046] The data tracing module is configured to upload the data in the storage node to the Merkle tree of the blockchain, determine whether the data in the storage node has been modified, and if so, generate an incremental snapshot and add it to the Merkle tree. Rollback operations can be performed based on the data collection time information;

[0047] The data security management module is configured to protect the data security of each data in the storage node based on dynamic desensitization technology and the national encryption SM4 algorithm.

[0048] In summary, this application includes at least one of the following beneficial technical effects:

[0049] 1. Multi-source compressed data is divided based on data type and data collection time information to obtain dynamic data sharding strategy information. Each storage node is then evaluated for health to select valid storage nodes. Based on the dynamic data sharding strategy information, each sub-shard data is distributed and stored on each valid storage node. A determination is made as to whether the storage cluster needs to be expanded. If expansion is required, it is performed using a data migration algorithm based on a consistent hash ring. By combining the dynamic sharding strategy with the consistent hash ring design, seamless expansion and efficient data migration of the storage cluster are achieved. During the expansion process, only approximately 1 / N of the data (N is the original number of nodes) needs to be migrated. This reduces data migration overhead compared to traditional expansion methods, enabling the system to easily cope with growing data volumes and business needs while ensuring high system availability and stability, thereby improving the efficiency of multi-source action data management.

[0050] 2. The average response time for spatial range queries using the spatiotemporal joint index is shortened compared to traditional methods, improving query efficiency and, in turn, the efficiency of multi-source action data management. Based on the metadata graph structure, relevant data nodes and their connections can be quickly located, providing strong support for deep data mining and comprehensive analysis, improving query efficiency, and further improving the efficiency of multi-source action data management.

[0051] 3. By combining dynamic desensitization technology and the national secret SM4 algorithm, privacy protection and efficiency optimization are given equal importance, providing strong guarantees for the secure storage and efficient use of multi-source motion data, thereby further improving the efficiency of multi-source motion data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a multi-source action data management method under an extensible distributed storage architecture, which is mainly reflected in this embodiment;

[0053] Figure 2 This is a schematic diagram of the process of step S3 in this embodiment;

[0054] Figure 3 This is a module diagram of a multi-source motion data management system under an extensible distributed storage architecture, which is mainly reflected in this embodiment.

[0055] Figure numerals: 1. Data access module; 2. Data preprocessing module; 3. Storage design module; 4. Service layer acquisition module; 5. Positioning query structure creation module; 6. Data tracing module; 7. Data security management module. DETAILED DESCRIPTION

[0056] The present application is further described in detail below with reference to the accompanying drawings.

[0057] The embodiments of the present application disclose a multi-source action data management method under a scalable distributed storage architecture.

[0058] A multi-source action data management method under a scalable distributed storage architecture includes the following steps:

[0059] Reference Figure 1 , step S1, obtains a multi-source data interface, performs an access operation on the original collected data information collected by the data acquisition module based on the multi-source data interface, and outputs a data access completion signal after the data access is completed. Step S1 specifically includes the following sub-steps:

[0060] Step S11 , obtaining a multi-source data interface and a data acquisition module, and establishing a signal connection link between the multi-source data interface and the data acquisition module, wherein the multi-source data interface is a unified access interface supporting multiple protocols.

[0061] Step S12: collecting information based on the data collection module to obtain original collected data information, where the original collected data information includes optical data information, IMU data information, and tactile data information.

[0062] In step S13, the raw collected data is accessed via the multi-source data interface, and a data access completion signal is outputted. The multi-source data interface uses the high-speed NVMe protocol for optical data access, ensuring fast and stable access to the raw optical data captured by the optical camera in the data acquisition module. The multi-source data interface uses the MQTT message queue for IMU data access, enabling efficient data transmission and asynchronous processing to meet the high-frequency data acquisition requirements of IMU data. The multi-source data interface uses gRPC streaming for tactile data access, ensuring real-time and accuracy.

[0063] Reference Figure 1 In step S2, after receiving the data access completion signal, the original collected data information is compressed to obtain multi-source compressed data information, the multi-source compressed data information is timestamp-calibrated, and the collection parameters of each data in the multi-source compressed data information are marked. Step S2 specifically includes the following sub-steps:

[0064] Step S21: After receiving the data access completion signal, the optical data information is compressed to obtain optical compressed data information, the IMU data information is compressed to obtain IMU compressed data information, and the tactile data information is compressed to obtain tactile compressed data information. The optical compressed data information, IMU compressed data information and tactile compressed data information are combined to form multi-source compressed data information.

[0065] Specifically, the LZ4 lossless compression algorithm is used in the embodiment of the present application. The LZ4 lossless compression algorithm has an efficient compression ratio greater than or equal to 50%, which can significantly reduce the storage space occupied by data and reduce the data transmission bandwidth requirement without losing data information.

[0066] Step S22, obtaining the collection time of each data in the optical compression data information, namely the optical data collection timestamp, obtaining the collection time of each data in the IMU compression data information, namely the IMU data collection timestamp, and obtaining the collection time of each data in the tactile compression data information, namely the tactile data collection timestamp.

[0067] In step S23 , the optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp are combined to form data acquisition time information.

[0068] Step S24 , performing timestamp calibration on the optical compressed data information, the IMU compressed data information, and the tactile data compressed information according to the optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp, and outputting a timestamp calibration completion signal after calibration.

[0069] In the embodiment of the present application, the data is timestamped to ensure consistency in the time dimension of data from different sources, with a calibration accuracy of ±0.5ms, providing an accurate time reference for subsequent data fusion and analysis.

[0070] Step S25: After receiving the timestamp calibration completion signal, the data type information, device ID information, and spatial coordinate information of each data in the optical compression data information, IMU compression data information, and tactile compression data information are judged and marked to obtain associated parameter marking information.

[0071] Among them, data type information includes optical type data, IMU type data and tactile type data, device ID information refers to the number of the acquisition device in the data acquisition module, and spatial coordinate information refers to the position coordinates of the acquisition point in space when each data is collected.

[0072] Reference Figure 1 and Figure 2 In step S3, data dynamic sharding strategy information is obtained based on the data type and acquisition time of each data in the multi-source compressed data information, a storage cluster is obtained and the health of each storage node in the storage cluster is determined, valid storage nodes are selected, and each data is dispersed and stored to the valid storage nodes based on the data dynamic sharding strategy information. It is determined whether the storage cluster needs to be expanded. If expansion is required, expansion operations are performed based on the data migration algorithm. Step S3 specifically includes the following sub-steps:

[0073] Step S31: preliminarily divide the multi-source compressed data information into a plurality of sub-slice data according to the data type information to obtain preliminary data slice information.

[0074] Specifically, the multi-source compressed data is initially divided into optical data, IMU data, and tactile data based on the data type information. The optical compressed data is stored using the Parquet columnar format, an efficient columnar storage format designed to optimize the storage and processing of large datasets. The IMU compressed data is stored using InfluxDB, an open-source time series database designed for processing metrics, events, and real-time analytics, enabling efficient storage and querying of time series data.

[0075] Step S32: The preliminary data sharding information is further divided into a plurality of sub-shards according to the data collection time information to obtain data dynamic sharding strategy information.

[0076] Specifically, based on the data collection time information, the collection time of each data in the preliminary data sharding information is divided into hours to obtain multiple data buckets, and then the collection time of each data in each data bucket is divided again into ten minutes to obtain the data dynamic sharding strategy information.

[0077] Step S33: obtain a storage cluster including multiple storage nodes, detect the health of each storage node, and if the health of a storage node is low, mark the storage node as an invalid storage node and mark all other storage nodes as valid storage nodes.

[0078] Specifically, according to the node health scoring formula The health of each storage node is scored to obtain the node health score S of each storage node, where w1 is the first weight ratio, w2 is the second weight ratio, and w3 is the third weight ratio. The value of W1 is 0.4, the value of W2 is 0.3, and the value of W3 is 0.3. %us refers to the CPU utilization, wa% refers to the disk IO wait, and R refers to the remaining network bandwidth.

[0079] The node health score S of each storage node is compared with the preset node health score threshold S'. If the node health score S of a storage node is less than or equal to the preset node health score threshold S', the storage node is marked as an invalid storage node, and the distribution of sharding tasks to the invalid storage node is suspended when storing data shards.

[0080] In this embodiment, the health score of each storage node is evaluated, and nodes with low scores will be suspended from accepting new sharding tasks to prevent them from affecting the performance of the entire system due to excessive load. Through this dynamic evaluation mechanism, nodes with uneven load can be discovered and handled in a timely manner, ensuring stable operation of the system.

[0081] Step S34: based on the data dynamic sharding strategy information, the sub-shard data is dispersedly stored on each valid storage node.

[0082] Step S35: Based on the currently known storage capacity and the total storage capacity of the storage cluster, determine whether the storage cluster needs to add new storage nodes for expansion. If expansion is required, dynamic expansion is performed based on the data migration algorithm of the consistent hash ring.

[0083] A consistent hash ring is a variant of a distributed hash table that evenly maps data onto a virtual hash ring. When a new node joins, only approximately 1 / N of the data needs to be migrated (N is the number of existing nodes), and the migration process maintains service availability. This localized data migration approach significantly reduces migration overhead and system impact, improving system scalability and flexibility.

[0084] Step S36: Based on the access frequency and timeliness of each data in the multi-source compressed data information, each data in the multi-source compressed data information is divided into cold and hot data to obtain a cold and hot data division result. The cold and hot data division result includes cold data and hot data, and the cold data is further compressed based on the erasure code technology.

[0085] In actual application, in the embodiment of the present application, the multi-source compressed data information is divided according to the data type information and the data collection time information to obtain the data dynamic sharding strategy information, and then the health score of each storage node is performed to select the valid storage node. According to the data dynamic sharding strategy information, the sub-shard data is dispersed and stored in each valid storage node, and it is determined whether expansion is needed. If expansion is required, the data migration algorithm based on the consistent hash ring is combined for expansion. By combining the dynamic sharding strategy with the consistent hash ring design, the imperceptible expansion and efficient data migration of the storage cluster are achieved. During the expansion process, only about 1 / N of the data needs to be migrated (N is the original number of nodes), which reduces the data migration overhead by about 70% compared with the traditional expansion method, allowing the system to easily cope with the growing data volume and business needs, while ensuring the high availability and stability of the system, thereby improving the efficiency of multi-source action data management.

[0086] Reference Figure 1 , step S4, obtains and performs query operations and read operations on each data in the storage cluster based on the user service layer. Step S4 specifically includes the following sub-steps:

[0087] Step S41, obtaining a user service layer, which includes a query engine and a machine learning interface.

[0088] Step S42: creating a signal connection link between the query engine and the storage cluster. Based on the query engine, the user inputs a query instruction, and the query engine filters and displays the data in the storage cluster that meets the query instruction.

[0089] Specifically, the query engine in the embodiment of the present application supports multiple query languages ​​and methods, including SQL, GraphQL, and custom domain-specific languages ​​(DSLs). Users can submit query requests through simple query statements or graphical interfaces. The query engine has intelligent parsing capabilities and can automatically parse multimodal correlation queries. For example, if the user enters a query instruction such as "retrieve all frames with a hand grip force greater than 20N and an elbow angle less than 90 degrees," the query engine automatically performs correlation analysis on the involved optical data, IMU data, and tactile data, and quickly and accurately returns a result set that meets the conditions, greatly simplifying the user's operation process and improving query efficiency and user experience.

[0090] Step S43: Create a signal connection link between the machine learning interface and the storage cluster, and read the data in the storage cluster based on the machine learning interface.

[0091] Specifically, the machine learning interface in this application supports mainstream machine learning frameworks such as TensorFlow and PyTorch, allowing users to directly read data from the storage cluster and use it as input for model training. In practice, users can integrate advanced machine learning algorithms into their own applications to conduct various research and development tasks such as motion recognition, behavioral analysis, and biomechanical modeling, expanding the application scenarios and value of the system.

[0092] Reference Figure 1 In step S5, an R* tree index structure is constructed based on each data in the storage node and a metadata map structure is created. The R* tree index structure and the metadata map structure are combined to form a data location query structure. Step S5 specifically includes the following sub-steps:

[0093] Step S51 : constructing an R*-tree index structure for the data collection time information of each data in the storage node and the spatial coordinate information in the associated parameter tag information.

[0094] Specifically, an R*-tree index is constructed for each frame of data. The R*-tree index structure is a commonly used spatial index structure. By calculating the bounding box and time range of each frame of data, extracting the coordinates of all marker points for each frame of optical data, and calculating the minimum outer bounding cube as the index key, the data in the multi-dimensional space is hierarchically organized and managed. When performing spatial range queries, the R*-tree index can quickly locate the area that may contain the target data based on the index key. For example, if you query "all data in the left-hand area between t1 and t2", the R*-tree index structure will perform the query and display all data in the left-hand area between t1 and t2. This greatly reduces the unnecessary data scanning range and speeds up the query process. Through experimental comparison, the average response time of spatial range queries using the spatiotemporal joint index is shortened by about 70% compared to traditional methods, which improves query efficiency and thus improves the efficiency of multi-source motion data management.

[0095] Step S52: Create a metadata graph structure based on the Neo4j graph database. The metadata graph structure includes multiple nodes and edges. The nodes include device ID nodes, data type nodes, and collection time nodes. The edges represent the relationship between the data in the storage nodes.

[0096] Specifically, the metadata graph structure refers to a relationship graph between device ID, data type, and collection time. This structure can intuitively display the complex relationships between multimodal data and support multi-hop correlation queries. For example, a user can query "a certain type of data collected by a certain device during a certain time period." The metadata graph structure quickly locates the relevant data nodes and their connections, providing strong support for in-depth data mining and comprehensive analysis, improving query efficiency and, in turn, the efficiency of multi-source action data management.

[0097] In step S53, the R*-tree index structure and the metadata graph structure are combined to form a data location query structure.

[0098] Reference Figure 1 In step S6, the data in the storage node is uploaded to the Merkle tree of the blockchain. It is determined whether the data in the storage node has been modified. If modified, an incremental snapshot is generated and added to the Merkle tree. A rollback operation can be performed based on the data collection time information. Step S6 specifically includes the following sub-steps:

[0099] Step S61: Upload the data in each storage node of the storage cluster to the Merkle tree of the blockchain, where the leaf nodes of the Merkle tree store the hash value of each data, and the non-leaf nodes store the hash value of their child nodes.

[0100] Step S62: determine whether the data in each storage node of the storage cluster has been modified. If modified, generate an incremental snapshot after the modification, and add the incremental snapshot to the Merkle tree.

[0101] In step S63, if the user encounters data errors or needs to restore historical data, the data can be rolled back based on the data collection time information, which is convenient for the user to restore when data errors occur or historical data needs to be restored. Through strict data version management, the reliability and traceability of the data are ensured.

[0102] Reference Figure 1 In step S7, the data security of each data in the storage node is protected according to the dynamic desensitization technology and the national secret SM4 algorithm. Step S7 specifically includes the following sub-steps:

[0103] Step S71: Process each data in the storage node in real time according to the dynamic desensitization technology, and add irreversible noise to each data in the storage node to meet the K-anonymity ≥ 5.

[0104] Step S72, obtain the hardware security module, and encrypt and store the cold data based on the national secret SM4 algorithm, where the key is hosted by the hardware security module, ensuring the security and reliability of the data. Combined with dynamic desensitization technology and the national secret SM4 algorithm, privacy protection and efficiency optimization are given equal importance, providing strong guarantees for the secure storage and efficient use of multi-source action data, thereby improving the management efficiency of multi-source action data.

[0105] The embodiments of the present application also disclose a multi-source action data management system under an extensible distributed storage architecture.

[0106] Reference Figure 3 ,The multi-source action data management system under the scalable distributed storage architecture includes:

[0107] The data access module is configured to obtain a multi-source data interface, perform an access operation on the original collected data information collected by the data acquisition module based on the multi-source data interface, and output a data access completion signal after the data access is completed.

[0108] The data preprocessing module is configured to, upon receiving a data access completion signal, perform a compression operation on the original collected data information to obtain multi-source compressed data information, perform time stamp calibration on the multi-source compressed data information, and mark the collection parameters of each data in the multi-source compressed data information.

[0109] The storage design module is configured to obtain data dynamic sharding strategy information based on the data type and collection time of each data in the multi-source compressed data information, obtain the storage cluster and determine the health of each storage node in the storage cluster, screen the valid storage nodes, and store each data in the valid storage nodes based on the data dynamic sharding strategy information, determine whether the storage cluster needs to be expanded, and if expansion is required, perform expansion operations based on the data migration algorithm.

[0110] The service layer acquisition module is configured to acquire and perform query operations and read operations on various data in the storage cluster based on the user service layer.

[0111] The positioning query structure creation module is configured to build an R* tree index structure based on the data in the storage node and create a metadata map structure. The R* tree index structure and the metadata map structure are combined to form a data positioning query structure.

[0112] The data tracing module is configured to upload the data in the storage node to the Merkle tree of the blockchain, determine whether the data in the storage node has been modified, and if so, generate an incremental snapshot and add it to the Merkle tree. Rollback operations can be performed based on the data collection time information.

[0113] The data security management module is configured to protect the data security of each data in the storage node based on dynamic desensitization technology and the national encryption SM4 algorithm.

[0114] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A multi-source action data management method under a scalable distributed storage architecture, characterized in that: The following steps are involved: Obtain a multi-source data interface, perform access operations on the original data information collected by the data acquisition module based on the multi-source data interface, and output a data access completion signal after data access is completed; After receiving the data access completion signal, the original collected data information is compressed to obtain multi-source compressed data information, the multi-source compressed data information is time-stamped, and the collection parameters of each data in the multi-source compressed data information are marked; The data dynamic sharding strategy information is obtained based on the data type and collection time of each data in the multi-source compressed data information. The storage cluster is obtained and the health of each storage node in the storage cluster is determined. The valid storage nodes are selected and the data is dispersed and stored to the valid storage nodes based on the data dynamic sharding strategy information. The storage cluster is determined to determine whether expansion is required. If expansion is required, the expansion operation is performed based on the data migration algorithm. Obtain and perform query and read operations on data in the storage cluster based on the user service layer; Construct an R* tree index structure based on each data in the storage node and create a metadata map structure, wherein the R* tree index structure and the metadata map structure are combined to form a data location query structure; Upload the data in the storage node to the Merkle tree of the blockchain to determine whether the data in the storage node has been modified. If modified, generate an incremental snapshot and add it to the Merkle tree. Rollback operations can be performed based on the data collection time information; The data security of each data in the storage node is protected based on dynamic desensitization technology and the national encryption SM4 algorithm.

2. The multi-source action data management method under the scalable distributed storage architecture according to claim 1 is characterized in that: The steps of obtaining a multi-source data interface, accessing the original collected data information collected by the data acquisition module based on the multi-source data interface, and outputting a data access completion signal after the data access is completed specifically include: Acquire a multi-source data interface and a data acquisition module, and establish a signal connection link between the multi-source data interface and the data acquisition module, wherein the multi-source data interface is a unified access interface supporting multiple protocols; Acquire original collected data information based on the data acquisition module, wherein the original collected data information includes optical data information, IMU data information, and tactile data information; The original collected data information is accessed based on the multi-source data interface, and a data access completion signal is output after the access is completed. The multi-source data interface adopts the NVMe protocol when accessing optical data information, the multi-source data interface adopts the MQTT message queue when accessing IMU data information, and the multi-source data interface adopts the gRPC stream when accessing tactile data information.

3. The multi-source action data management method under the scalable distributed storage architecture according to claim 2 is characterized in that: The steps of compressing the original collected data information to obtain multi-source compressed data information after receiving the data access completion signal, calibrating the timestamp of the multi-source compressed data information, and marking the collection parameters of each data in the multi-source compressed data information specifically include: After receiving the data access completion signal, compressing the optical data information to obtain optical compressed data information, compressing the IMU data information to obtain IMU compressed data information, and compressing the tactile data information to obtain tactile compressed data information, wherein the optical compressed data information, the IMU compressed data information, and the tactile compressed data information are combined to form multi-source compressed data information; Obtaining the acquisition time of each data in the optical compression data information, namely, the optical data acquisition timestamp; obtaining the acquisition time of each data in the IMU compression data information, namely, the IMU data acquisition timestamp; obtaining the acquisition time of each data in the tactile compression data information, namely, the tactile data acquisition timestamp; The optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp are combined to form data acquisition time information; Performing time stamp calibration on the optical compressed data information, the IMU compressed data information, and the tactile data compressed data information according to the optical data acquisition time stamp, the IMU data acquisition time stamp, and the tactile data acquisition time stamp, and outputting a time stamp calibration completion signal after calibration; After receiving the timestamp calibration completion signal, the data type information, device ID information, and spatial coordinate information of each data in the optical compression data information, IMU compression data information, and tactile compression data information are judged and marked to obtain associated parameter marking information.

4. The multi-source action data management method under the scalable distributed storage architecture according to claim 3 is characterized in that: The steps of obtaining data dynamic sharding strategy information based on the data type and acquisition time of each data in the multi-source compressed data information, obtaining a storage cluster and determining the health of each storage node in the storage cluster, screening effective storage nodes, distributing and storing each data to the effective storage nodes based on the data dynamic sharding strategy information, determining whether the storage cluster needs to be expanded, and if expansion is required, performing expansion operations based on a data migration algorithm specifically include: Preliminarily dividing the multi-source compressed data information into a plurality of sub-shard data according to the data type information to obtain preliminary data shard information; According to the data collection time information, the data initial sharding information is further divided into multiple sub-sharding data to obtain data dynamic sharding strategy information; Acquire a storage cluster, the storage cluster including multiple storage nodes, detect the health of each storage node, and if the health of a storage node is low, mark the storage node as an invalid storage node, and mark all other storage nodes except the invalid storage node as valid storage nodes; Based on the data dynamic sharding strategy information, each sub-shard data is dispersed and stored on each valid storage node; Based on the currently known storage capacity and the total storage capacity of the storage cluster, determine whether the storage cluster needs to add new storage nodes for expansion. If expansion is required, perform dynamic expansion based on the data migration algorithm of the consistent hash ring; According to the access frequency and timeliness of each data in the multi-source compressed data information, each data in the multi-source compressed data information is divided into cold and hot data layers to obtain a cold and hot data division result. The cold and hot data division result includes cold data and hot data, and the cold data is further compressed based on the erasure code technology.

5. The multi-source action data management method under the scalable distributed storage architecture according to claim 4 is characterized in that: The steps of obtaining and performing query operations and reading operations on each data in the storage cluster based on the user service layer specifically include: Obtaining a user service layer, wherein the user service layer includes a query engine and a machine learning interface; A signal connection link is established between the query engine and the storage cluster. Based on the query engine, a user inputs a query instruction, and the query engine filters and displays data in the storage cluster that meets the query instruction. A signal connection link is created between the machine learning interface and the storage cluster, and data in the storage cluster is read based on the machine learning interface.

6. The multi-source action data management method under the scalable distributed storage architecture according to claim 5, characterized in that: The steps of constructing an R* tree index structure based on each data in the storage node and creating a metadata map structure, and combining the R* tree index structure and the metadata map structure to form a data location query structure specifically include: Construct an R* tree index structure for the data collection time information of each data in the storage node and the spatial coordinate information in the associated parameter tag information; Creating a metadata graph structure based on a Neo4j graph database. The metadata graph structure includes multiple nodes and edges. The nodes include device ID nodes, data type nodes, and collection time nodes. The edges represent the relationship between the data in the storage nodes. The R*-tree index structure and the metadata graph structure are combined to form a data location query structure.

7. The multi-source action data management method under the scalable distributed storage architecture according to claim 6, characterized in that: The steps of uploading the data in the storage node to the Merkle tree of the blockchain, determining whether the data in the storage node has been modified, and if so, generating an incremental snapshot and adding it to the Merkle tree, and performing a rollback operation based on the data collection time information, specifically include: Upload the data in each storage node of the storage cluster to the Merkle tree of the blockchain. The leaf nodes of the Merkle tree store the hash value of each data, and the non-leaf nodes store the hash value of their child nodes. Determine whether the data in each storage node of the storage cluster has been modified. If so, generate an incremental snapshot after the modification and add the incremental snapshot to the Merkle tree. If the user encounters data errors or needs to restore historical data, the data can be rolled back based on the data collection time information.

8. The multi-source action data management method under the scalable distributed storage architecture according to claim 7, characterized in that: The steps to protect the data security of each data in the storage node based on dynamic desensitization technology and the national encryption SM4 algorithm include: The data in the storage nodes are processed in real time based on dynamic desensitization technology, and irreversible noise is added to the data in the storage nodes to meet the K-anonymity ≥ 5; Obtain a hardware security module and encrypt and store cold data based on the national secret SM4 algorithm, where the key is managed by the hardware security module.

9. A multi-source motion data management system under a scalable distributed storage architecture, characterized in that: The multi-source action data management system under the scalable distributed storage architecture is used to implement the multi-source action data management method under the scalable distributed storage architecture described in any one of claims 1 to 8, comprising: A data access module is configured to obtain a multi-source data interface, access the original data information collected by the data acquisition module based on the multi-source data interface, and output a data access completion signal after the data access is completed; The data preprocessing module is configured to, upon receiving a data access completion signal, compress the original collected data information to obtain multi-source compressed data information, perform time stamp calibration on the multi-source compressed data information, and mark the collection parameters of each data in the multi-source compressed data information; The storage design module is configured to obtain data dynamic sharding strategy information based on the data type and collection time of each data in the multi-source compressed data information, obtain the storage cluster and determine the health of each storage node in the storage cluster, select valid storage nodes, and disperse the data to the valid storage nodes based on the data dynamic sharding strategy information. It determines whether the storage cluster needs to be expanded. If expansion is required, expansion is performed based on the data migration algorithm. The service layer acquisition module is configured to acquire and perform query operations and read operations on various data in the storage cluster based on the user service layer; A location query structure creation module is configured to construct an R* tree index structure based on each data in the storage node and create a metadata map structure, wherein the R* tree index structure and the metadata map structure are combined to form a data location query structure; The data tracing module is configured to upload the data in the storage node to the Merkle tree of the blockchain, determine whether the data in the storage node has been modified, and if so, generate an incremental snapshot and add it to the Merkle tree. Rollback operations can be performed based on the data collection time information; The data security management module is configured to protect the data security of each data in the storage node based on dynamic desensitization technology and the national encryption SM4 algorithm.

Images