Multi-source action data management system and method under extensible distributed storage architecture

Abstract

The invention discloses a multi-source action data management system and method under an extensible distributed storage architecture, and relates to the technical field of data management, and the method comprises the steps: accessing original collection data information collected by a data collection module based on a multi-source data interface, and outputting a data access completion signal; compressing the original collected data information to obtain multi-source compressed data information, calibrating a timestamp of the multi-source compressed data information, and marking collection parameters of data in the multi-source compressed data information; data dynamic fragmentation strategy information is obtained according to the data type of the multi-source compressed data information and acquisition time division, the health degree of each storage node in a storage cluster is judged, effective storage nodes are screened, data are dispersed and stored to the effective storage nodes based on the data dynamic fragmentation strategy information, and whether the storage cluster needs to be expanded or not is judged. If expansion is needed, expansion is carried out based on a data migration algorithm. The multi-source action data management method has the effect of improving the multi-source action data management efficiency.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to a multi-source action data management system and method under an extensible distributed storage architecture. Background Art

[0002] Currently, with the continuous advancement of the informatization process, data has become one of the most important assets in modern society. The diversification of data sources and the explosive growth of data volume make data management particularly important. Multi-source action data management is a key technology for constructing a comprehensive data view, which can help with unified management, analysis, and application in large-scale action capture scenarios. Therefore, multi-source action data management under an extensible distributed storage architecture is particularly important.

[0003] Existing multi-source action data management methods refer to data acquisition and integration, data preprocessing, storage scheme design, metadata management, data storage implementation, and index and retrieval optimization. However, in the existing multi-source action data management methods, during the data acquisition process, the data sources are extremely diverse, the data storage formats are different, and the data dimensions vary greatly. As a result, during the data processing process, it is difficult to accurately match and fuse different modalities of data. Moreover, in actual action capture scenarios, the peak value of multi-camera synchronous writing can reach 10 Gbps, and traditional distributed systems are difficult to bear such high writing pressure, easily resulting in data congestion and loss, leading to low efficiency of multi-source action data management and there is room for improvement. Summary of the Invention

[0004] In order to improve the efficiency of multi-source action data management, this application provides a multi-source action data management system and method under an extensible distributed storage architecture.

[0005] In a first aspect, the multi-source action data management method under an extensible distributed storage architecture provided by this application adopts the following technical solution: The multi-source action data management method under an extensible distributed storage architecture includes: Obtain a multi-source data interface, and perform an access operation on the original acquisition data information collected by the data acquisition module based on the multi-source data interface. After the data access is completed, a data access completion signal is output; When the data access completion signal is received, perform a compression operation on the original acquisition data information to obtain multi-source compressed data information, perform timestamp calibration on the multi-source compressed data information, and mark the acquisition parameters of each data in the multi-source compressed data information; Divide according to the data types and collection times of the data in the multi-source compressed data information to obtain data dynamic sharding strategy information, obtain the storage cluster and judge the health of each storage node in the storage cluster, filter out valid storage nodes, and disperse and store each data to the valid storage nodes based on the data dynamic sharding strategy information. Judge whether the storage cluster needs to be expanded. If expansion is required, perform an expansion operation based on the data migration algorithm; Obtain and perform query operations and read operations on each data in the storage cluster based on the user service layer; Construct an R* tree index structure and create a metadata graph structure based on each data in the storage node. The R* tree index structure and the metadata graph structure are combined to form a data location query structure; Upload the data in the storage node to the Merkle tree of the blockchain, judge whether the data in the storage node is modified. If it is modified, generate an incremental snapshot and add it to the Merkle tree. A rollback operation can be performed based on the data collection time information; Protect the data security of each data in the storage node according to the dynamic desensitization technology and the national cipher SM4 algorithm.

[0006] Preferably, obtain a multi-source data interface and a data collection module, and establish a signal connection link between the multi-source data interface and the data collection module. Among them, the multi-source data interface is a unified access interface that supports multiple protocols; Collect information based on the data collection module to obtain original collection data information, and the original collection data information includes optical data information, IMU data information, and tactile data information; Perform an access operation on the original collection data information based on the multi-source data interface. After the access is completed, output a data access completion signal. Among them, the multi-source data interface uses the NVMe protocol when accessing optical data information, and the multi-source data interface uses the MQTT message queue when accessing IMU data information. The multi-source data interface uses the gRPC stream method to access tactile data information.

[0007] Preferably, after receiving the data access completion signal, compress the optical data information to obtain optical compressed data information, compress the IMU data information to obtain IMU compressed data information, and compress the tactile data information to obtain tactile compressed data information. The optical compressed data information, the IMU compressed data information, and the tactile compressed data information are combined to form multi-source compressed data information; Obtain the collection time of each data in the optical compressed data information, that is, the optical data collection timestamp, obtain the collection time of each data in the IMU compressed data information, that is, the IMU data collection timestamp, and obtain the collection time of each data in the tactile compressed data information, that is, the tactile data collection timestamp; The optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp are combined to form data acquisition time information; According to the optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp, timestamp calibration is performed on the optical compressed data information, the IMU compressed data information, and the tactile data compressed information, and a timestamp calibration completion signal is output after calibration; After receiving the timestamp calibration completion signal, the data type information, device ID information, and spatial coordinate information of each data in the optical compressed data information, the IMU compressed data information, and the tactile compressed data information are judged and marked to obtain associated parameter marking information.

[0008] Preferably, the multi-source compressed data information is initially divided into multiple sub-shard data according to the data type information to obtain data initial sharding information; The data initial sharding information is further divided into multiple sub-shard data according to the data acquisition time information to obtain a data dynamic sharding strategy information; A storage cluster is obtained. The storage cluster includes multiple storage nodes. The health of each storage node is detected. If the health of a storage node is low, the storage node is marked as an invalid storage node, and the other storage nodes except the invalid storage nodes are marked as valid storage nodes; Based on the data dynamic sharding strategy information, each sub-shard data is dispersedly stored on each valid storage node; According to the currently known storage capacity and the total storage capacity of the storage cluster, it is judged whether the storage cluster needs to add new storage nodes for expansion. If expansion is required, dynamic expansion is performed based on the data migration algorithm of the consistent hashing ring; According to the access frequency and timeliness of each data in the multi-source compressed data information, each data in the multi-source compressed data information is stratified into cold and hot data to obtain a cold and hot data division result. The cold and hot data division result includes cold data and hot data, and further compression is performed on the cold data based on the erasure code technology.

[0009] Preferably, a user service layer is obtained. The user service layer includes a query engine and a machine learning interface; A signal connection link is created between the query engine and the storage cluster. Based on the query engine, when a user inputs a query instruction, the query engine filters and displays the data in the storage cluster that meets the query instruction; A signal connection link is created between the machine learning interface and the storage cluster, and each data in the storage cluster is read based on the machine learning interface.

[0010] Preferably, construct an R*-tree index structure for the data acquisition time information of each data in the storage node and the spatial coordinate information in the associated parameter marking information; Create a metadata graph structure according to the Neo4j graph database. The metadata graph structure includes multiple nodes and edges. The nodes include device ID nodes, data type nodes, and acquisition time nodes. The edges represent the relationships between the data in the storage nodes; The R*-tree index structure and the metadata graph structure are combined to form a data location query structure.

[0011] Preferably, upload the data in each storage node of the storage cluster to the Merkle tree of the blockchain. Among them, the leaf nodes of the Merkle tree store the hash values of each data, and the non-leaf nodes store the hash values of their child nodes; Judge whether the data in each storage node of the storage cluster is modified. If it is modified, generate an incremental snapshot after the modification and add the incremental snapshot to the Merkle tree; If the user has data errors or needs to restore historical data, a rollback operation can be performed on the data based on the data acquisition time information.

[0012] Preferably, perform real-time processing on each data in the storage node according to the dynamic desensitization technology, and add irreversible noise to each data in the storage node to satisfy K-anonymity ≥ 5; Obtain a hardware security module and encrypt and store the cold data based on the national secret SM4 algorithm, where the key is entrusted by the hardware security module.

[0013] In a second aspect, the present application provides a multi-source action data management system under an extensible distributed storage architecture, adopting the following technical solutions: A multi-source action data management system under an extensible distributed storage architecture, including: A data access module configured to obtain a multi-source data interface and perform an access operation on the original acquisition data information collected by the data acquisition module based on the multi-source data interface. After the data access is completed, a data access completion signal is output; A data preprocessing module configured to, when receiving the data access completion signal, perform a compression operation on the original acquisition data information to obtain multi-source compressed data information, calibrate the timestamp of the multi-source compressed data information, and mark the acquisition parameters of each data in the multi-source compressed data information; A storage design module, configured to divide the multi-source compressed data information according to the data type and collection time of each data to obtain data dynamic sharding policy information, obtain a storage cluster, judge the health of each storage node in the storage cluster, filter out effective storage nodes, and disperse each data to the effective storage nodes based on the data dynamic sharding policy information, judge whether the storage cluster needs to be expanded, and if so, perform an expansion operation based on a data migration algorithm; A service layer acquisition module, configured to acquire and perform query operations and read operations on each data in the storage cluster based on the user service layer; A location query structure creation module, configured to construct an R* tree index structure and create a metadata graph structure based on each data in the storage node, and the R* tree index structure and the metadata graph structure are combined to form a data location query structure; A data traceability module, configured to upload the data in the storage node to the Merkle tree of the blockchain, judge whether the data in the storage node is modified, if so, generate an incremental snapshot and add it to the Merkle tree, and a rollback operation can be performed based on the data collection time information; A data security management module, configured to protect the data security of each data in the storage node according to the dynamic desensitization technology and the national secret SM4 algorithm.

[0014] In summary, the present application includes at least one of the following beneficial technical effects: 1. The multi-source compressed data information is divided according to the data type information and the data collection time information to obtain data dynamic sharding policy information, then the health of each storage node is scored to select effective storage nodes, each sub-sharded data is dispersed and stored in each effective storage node according to the data dynamic sharding policy information, and it is judged whether the storage cluster needs to be expanded. If so, expansion is performed in combination with a data migration algorithm based on a consistent hashing ring. Through the combination of the dynamic sharding strategy and the consistent hashing ring design, seamless expansion of the storage cluster and efficient data migration are achieved. During the expansion process, only about 1 / N of the data needs to be migrated (N is the number of original nodes), reducing the data migration overhead compared with traditional expansion methods, enabling the system to easily handle the growing data volume and business requirements, while ensuring the high availability and stability of the system, thereby improving the management efficiency of multi-source action data; 2. The average response time for spatial range queries using the spatio-temporal joint index is shortened compared with traditional methods, improving the query efficiency, and thus improving the management efficiency of multi-source action data. According to the metadata graph structure, the relevant data nodes and their connection relationships can be quickly located, providing strong support for in-depth mining and comprehensive analysis of data, improving the query efficiency, and further improving the management efficiency of multi-source action data; 3. By combining dynamic data masking technology and the national cryptographic SM4 algorithm, it emphasizes both privacy protection and efficiency optimization, providing strong guarantee for the secure storage and efficient utilization of multi-source action data, and further improving the management efficiency of multi-source action data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic flowchart of the multi-source action data management method under the extensible distributed storage architecture mainly embodied in this embodiment; Figure 2 It is a schematic flowchart mainly embodying step S3 in this embodiment; Figure 3 It is a schematic block diagram of the multi-source action data management system under the extensible distributed storage architecture mainly embodied in this embodiment.

[0016] Reference numerals: 1, data access module; 2, data preprocessing module; 3, storage design module; 4, service layer acquisition module; 5, positioning query structure creation module; 6, data traceability module; 7, data security management module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] The following further describes the present application in detail with reference to the accompanying drawings.

[0018] The embodiment of the present application discloses a multi-source action data management method under an extensible distributed storage architecture.

[0019] The multi-source action data management method under the extensible distributed storage architecture includes the following steps: Refer to Figure 1 , step S1, obtain a multi-source data interface, and perform an access operation on the original acquisition data information collected by the data acquisition module based on the multi-source data interface. After the data access is completed, a data access completion signal is output. Step S1 specifically includes the following sub-steps: Step S11, obtain a multi-source data interface and a data acquisition module, and establish a signal connection link between the multi-source data interface and the data acquisition module. Among them, the multi-source data interface is a unified access interface that supports multiple protocols.

[0020] Step S12, collect information based on the data acquisition module to obtain the original acquisition data information, and the original acquisition data information includes optical data information, IMU data information, and tactile data information.

[0021] Step S13, based on the multi-source data interface, perform an access operation on the original collected data information. After the access is completed, output a data access completion signal. Among them, when the multi-source data interface accesses the optical data information, it uses the high-speed NVMe protocol to ensure that the optical raw data information collected by the optical camera in the data acquisition module can be accessed quickly and stably. When the multi-source data interface accesses the IMU data information, it uses the MQTT message queue to achieve efficient data transmission and asynchronous processing to meet the requirements of high-frequency data acquisition of the IMU data information. When the multi-source data interface accesses the tactile data information, it uses the gRPC stream method for access to ensure the real-time and accuracy of the tactile data information.

[0022] Refer to Figure 1 , Step S2, when the data access completion signal is received, perform a compression operation on the original collected data information to obtain multi-source compressed data information, perform timestamp calibration on the multi-source compressed data information, and mark the acquisition parameters of each data in the multi-source compressed data information. Step S2 specifically includes the following sub-steps: Step S21, when the data access completion signal is received, compress the optical data information to obtain optical compressed data information, compress the IMU data information to obtain IMU compressed data information, compress the tactile data information to obtain tactile compressed data information. The optical compressed data information, IMU compressed data information, and tactile compressed data information are combined to form multi-source compressed data information.

[0023] Specifically, in the embodiment of the present application, the LZ4 lossless compression algorithm is adopted. Among them, the LZ4 lossless compression algorithm has an efficient compression ratio of greater than or equal to 50%, which can significantly reduce the storage space occupancy of data and reduce the data transmission bandwidth requirement without losing data information.

[0024] Step S22, obtain the acquisition time of each data in the optical compressed data information, that is, the optical data acquisition timestamp, obtain the acquisition time of each data in the IMU compressed data information, that is, the IMU data acquisition timestamp, and obtain the acquisition time of each data in the tactile compressed data information, that is, the tactile data acquisition timestamp.

[0025] Step S23, the optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp are combined to form data acquisition time information.

[0026] Step S24, according to the optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp, perform timestamp calibration on the optical compressed data information, the IMU compressed data information, and the tactile data compressed information. After calibration, output a timestamp calibration completion signal.

[0027] In the embodiments of the present application, by calibrating timestamps of data, the consistency of data from different sources in the time dimension is ensured, and the calibration accuracy reaches ±0.5 ms, providing an accurate time benchmark for subsequent data fusion and analysis.

[0028] Step S25: After receiving the signal indicating that the timestamp calibration is completed, judge and mark the data type information, device ID information, and spatial coordinate information of each data in the optical compression data information, IMU compression data information, and tactile compression data information to obtain the associated parameter marking information.

[0029] Among them, the data type information includes optical type data, IMU type data, and tactile type data. The device ID information refers to the number of the acquisition device in the data acquisition module, and the spatial coordinate information refers to the position coordinates of the acquisition point in space when each data is acquired.

[0030] Refer to Figure 1 and Figure 2 , Step S3: According to the data types and acquisition times of each data in the multi-source compression data information, divide to obtain the data dynamic sharding strategy information, obtain the storage cluster, judge the health of each storage node in the storage cluster, screen out the effective storage nodes, and disperse each data to the effective storage nodes based on the data dynamic sharding strategy information. Judge whether the storage cluster needs to be expanded. If expansion is required, perform an expansion operation based on the data migration algorithm. Step S3 specifically includes the following sub-steps: Step S31: According to the data type information, preliminarily divide the multi-source compression data information into multiple sub-sharded data to obtain the data preliminary sharding information.

[0031] Specifically, according to the optical type data, IMU type data, and tactile type data in the data type information, the multi-source compression information is preliminarily divided. Among them, the optical compression data information is stored in the Parquet columnar format. Parquet is an efficient columnar storage format designed to optimize the storage and processing of large-scale data sets. The IMU compression data information is stored using InfluxDB time series. InfluxDB is an open-source time series database dedicated to processing metrics, events, and real-time analysis, and can efficiently store and query time series data.

[0032] Step S32: According to the data acquisition time information, divide the data preliminary sharding information into multiple sub-sharded data again to obtain the data dynamic sharding strategy information.

[0033] Specifically, based on the data acquisition time information, divide the acquisition time of each data in the data preliminary sharding information by hour to obtain multiple data buckets, and then divide the acquisition time of each data in each data bucket by ten minutes again to obtain the data dynamic sharding strategy information.

[0034] Step S33: Obtain a storage cluster which includes multiple storage nodes. Detect the healthiness of each storage node. If the healthiness of a storage node is low, mark this storage node as an invalid storage node, and mark the other storage nodes except the invalid storage nodes as valid storage nodes.

[0035] Specifically, according to the node healthiness scoring formula score the healthiness of each storage node to obtain the node healthiness score S of each storage node. Among them, w1 is the first weight ratio, w2 is the second weight ratio, w3 is the third weight ratio, the value of W1 is 0.4, the value of W2 is 0.3, the value of W3 is 0.3, %us refers to the CPU utilization rate, wa% refers to the disk I / O waiting, and R refers to the remaining network bandwidth.

[0036] Compare the node healthiness score S of each storage node with the preset node healthiness score threshold S'. If the node healthiness score S of a storage node is less than or equal to the preset node healthiness score threshold S', mark this storage node as an invalid storage node, and suspend the distribution of the sharding task for the invalid storage node during data sharding storage.

[0037] In the embodiment of the present application, by evaluating the healthiness scores of each storage node, the nodes with lower scores will be suspended from receiving new sharding tasks to avoid affecting the performance of the entire system due to overloading. Through this dynamic evaluation mechanism, nodes with unbalanced loads can be discovered and processed in a timely manner to ensure the stable operation of the system.

[0038] Step S34: Disperse and store each sub-sharded data on each valid storage node based on the data dynamic sharding policy information.

[0039] Step S35: According to the currently known storage capacity and the total storage capacity of the storage cluster, determine whether the storage cluster needs to add new storage nodes for expansion. If expansion is required, perform dynamic expansion based on the data migration algorithm of the consistent hashing ring.

[0040] Among them, the consistent hashing ring is a variant of the distributed hash table, which evenly maps data to a virtual hash ring. When a new node joins, only about 1 / N of the data needs to be migrated (N is the number of original nodes), and the service remains available during the migration process. This local data migration method greatly reduces the data migration overhead and the impact on the system, and improves the scalability and flexibility of the system.

[0041] Step S36: According to the access frequency and timeliness of each data in the multi-source compressed data information, perform cold and hot data stratification on each data in the multi-source compressed data information to obtain a cold and hot data partitioning result. The cold and hot data partitioning result includes cold data and hot data, and further compress the cold data based on the erasure code technology.

[0042] In practical applications, in the embodiment of the present application, the multi-source compressed data information is partitioned through data type information and data collection time information to obtain data dynamic sharding policy information, and then the health scores of each storage node are evaluated to select effective storage nodes. Each sub-sharded data is dispersedly stored in each effective storage node according to the data dynamic sharding policy information, and it is judged whether expansion is required. If expansion is required, expansion is performed in combination with the data migration algorithm based on the consistent hashing ring. Through the combination of the dynamic sharding policy and the consistent hashing ring design, the seamless expansion of the storage cluster and efficient data migration are realized. During the expansion process, only about 1 / N of the data needs to be migrated (N is the number of original nodes), which reduces the data migration overhead by about 70% compared with the traditional expansion method, enables the system to easily cope with the growing data volume and business requirements, and at the same time ensures the high availability and stability of the system, thereby improving the management efficiency of multi-source motion data.

[0043] Refer to Figure 1 , Step S4: Obtain and perform query operations and read operations on each data in the storage cluster based on the user service layer. Step S4 specifically includes the following sub-steps: Step S41: Obtain the user service layer, and the user service layer includes a query engine and a machine learning interface.

[0044] Step S42: Create a signal connection link between the query engine and the storage cluster. Based on the query engine, the user inputs a query instruction, and the query engine filters and displays the data in the storage cluster that meets the query instruction.

[0045] Specifically, the query engine in the embodiment of the present application supports multiple query languages and methods, including SQL, GraphQL, and custom domain-specific languages (DSLs). Users can submit query requests through simple query statements or graphical interfaces. The query engine has intelligent parsing capabilities and can automatically parse multi-modal association queries. For example, if the query instruction input by the user is "retrieve all frames where the hand grasping force is greater than 20N and the elbow angle is less than 90 degrees", the query engine automatically performs association analysis on the involved optical data, IMU data, and tactile data, and quickly and accurately returns the result set that meets the conditions, greatly simplifying the user operation process and improving the query efficiency and user experience.

[0046] Step S43, create a signal connection link between the machine learning interface and the storage cluster, and read each piece of data in the storage cluster based on the machine learning interface.

[0047] Specifically, the machine learning interface in the embodiments of the present application supports mainstream machine learning frameworks such as TensorFlow and PyTorch, allowing users to directly read the data in the storage cluster and use it as the input for model training. In actual applications, users can integrate advanced machine learning algorithms into their own applications to conduct various research and development work such as action recognition, behavior analysis, and biomechanics modeling, expanding the application scenarios and value of the system.

[0048] Refer to Figure 1 , step S5, construct an R*-tree index structure based on each piece of data in the storage node and create a metadata graph structure. The R*-tree index structure and the metadata graph structure are combined to form a data location query structure. Step S5 specifically includes the following sub-steps: Step S51, construct an R*-tree index structure for the data acquisition time information of each piece of data in the storage node and the spatial coordinate information in the associated parameter marking information.

[0049] Specifically, construct an R*-tree index for each frame of data. The R*-tree index structure is a commonly used spatial index structure. By calculating the bounding box and time range of each frame of data, extract the coordinates of all marked points for each frame of optical data, calculate the minimum bounding cube and use it as the index key to hierarchically organize and manage the data in the multi-dimensional space. When performing a spatial range query, the R*-tree index can quickly locate the area that may contain the target data based on the index key. For example, when querying "all data in the left hand area between t1 - t2", the R*-tree index structure performs the query and displays all data in the left hand area between t1 - t2. This greatly reduces the unnecessary data scanning range and accelerates the query process. Through experimental comparison, the average response time for spatial range queries using the spatio-temporal joint index is shortened by about 70% compared to the traditional method, improving the query efficiency and thus improving the multi-source action data management efficiency.

[0050] Step S52, create a metadata graph structure according to the Neo4j graph database. The metadata graph structure includes multiple nodes and edges. The nodes include device ID nodes, data type nodes, and acquisition time nodes. The edges represent the relationships between each piece of data in the storage node.

[0051] Specifically, the metadata graph structure refers to the relationship graph of device ID - data type - acquisition time. The metadata graph structure can intuitively display the complex association relationships between multi-modal data and support multi-hop association queries. For example, users can query "a certain type of data collected by a certain device within a certain time period", and the metadata graph structure can quickly locate the relevant data nodes and their connection relationships, providing strong support for in-depth data mining and comprehensive analysis, improving the query efficiency, and thus improving the management efficiency of multi-source action data.

[0052] Step S53, the R* tree index structure and the metadata graph structure are combined to form a data location query structure.

[0053] Refer to Figure 1 , Step S6, upload the data in the storage node to the Merkle tree of the blockchain, determine whether the data in the storage node is modified. If it is modified, generate an incremental snapshot and add it to the Merkle tree. A rollback operation can be performed based on the data acquisition time information. Step S6 specifically includes the following sub-steps: Step S61, upload the data in each storage node of the storage cluster to the Merkle tree of the blockchain. Among them, the leaf nodes of the Merkle tree store the hash values of each data, and the non-leaf nodes store the hash values of their child nodes.

[0054] Step S62, determine whether the data in each storage node of the storage cluster is modified. If it is modified, generate an incremental snapshot after the modification and add the incremental snapshot to the Merkle tree.

[0055] Step S63, if the user has data errors or needs to restore historical data, a rollback operation can be performed on the data based on the data acquisition time information, which is convenient for users to recover when data errors occur or historical data needs to be restored. Through strict data version management, the reliability and traceability of the data are ensured.

[0056] Refer to Figure 1 , Step S7, protect the data security of each data in the storage node according to the dynamic data masking technology and the national cryptography SM4 algorithm. Step S7 specifically includes the following sub-steps: Step S71, perform real-time processing on each data in the storage node according to the dynamic data masking technology, and add irreversible noise to each data in the storage node to satisfy K-anonymity ≥ 5.

[0057] Step S72: Obtain a hardware security module, encrypt and store cold data based on the national cryptography SM4 algorithm, where the key is entrusted by the hardware security module, ensuring the security and reliability of the data. Combining dynamic desensitization technology and the national cryptography SM4 algorithm, both privacy protection and efficiency optimization are emphasized, providing a strong guarantee for the secure storage and efficient utilization of multi-source action data, thereby improving the management efficiency of multi-source action data.

[0058] The embodiment of the present application also discloses a multi-source action data management system under an extensible distributed storage architecture.

[0059] Refer to Figure 3 , the multi-source action data management system under the extensible distributed storage architecture includes: A data access module, configured to obtain a multi-source data interface, perform an access operation on the original collected data information collected by the data collection module based on the multi-source data interface, and output a data access completion signal after the data access is completed.

[0060] A data preprocessing module, configured to perform a compression operation on the original collected data information to obtain multi-source compressed data information when receiving the data access completion signal, perform timestamp calibration on the multi-source compressed data information, and mark the collection parameters of each data in the multi-source compressed data information.

[0061] A storage design module, configured to divide according to the data type and collection time of each data in the multi-source compressed data information to obtain data dynamic sharding policy information, obtain a storage cluster and judge the health of each storage node in the storage cluster, screen valid storage nodes, and disperse each data to the valid storage nodes based on the data dynamic sharding policy information, judge whether the storage cluster needs to be expanded, and if so, perform an expansion operation based on the data migration algorithm.

[0062] A service layer acquisition module, configured to obtain and perform query operations and read operations on each data in the storage cluster based on the user service layer.

[0063] A location query structure creation module, configured to construct an R* tree index structure and create a metadata graph structure based on each data in the storage node, and the R* tree index structure and the metadata graph structure are combined to form a data location query structure.

[0064] A data traceability module, configured to upload the data in the storage node to the Merkle tree of the blockchain, judge whether the data in the storage node is modified, if modified, generate an incremental snapshot and add it to the Merkle tree, and a rollback operation can be performed based on the data collection time information.

[0065] A data security management module, configured to protect the data security of each data in the storage node according to dynamic desensitization technology and the national cryptography SM4 algorithm.

[0066] The above are all preferred embodiments of the present application, and do not limit the protection scope of the present application accordingly. Therefore, all equivalent changes made according to the structure, shape, and principle of the present application shall be covered within the protection scope of the present application.

Claims

1. A multi-source action data management method under an extensible distributed storage architecture, characterized in that, It includes the following steps: Obtain a multi-source data interface, and perform an access operation on the original collected data information collected by the data collection module based on the multi-source data interface. After the data access is completed, a data access completion signal is output; When the data access completion signal is received, perform a compression operation on the original collected data information to obtain multi-source compressed data information, perform timestamp calibration on the multi-source compressed data information, and mark the acquisition parameters of each data in the multi-source compressed data information; According to the data types and acquisition times of each data in the multi-source compressed data information, divide to obtain data dynamic sharding strategy information, obtain a storage cluster and judge the health of each storage node in the storage cluster, filter out effective storage nodes, and disperse each data to the effective storage nodes based on the data dynamic sharding strategy information. Judge whether the storage cluster needs to be expanded. If expansion is required, perform an expansion operation based on the data migration algorithm; Obtain and perform query operations and read operations on each data in the storage cluster based on the user service layer; Construct an R* tree index structure and create a metadata graph structure based on each data in the storage node. The R* tree index structure and the metadata graph structure are combined to form a data location query structure; Upload the data in the storage node to the Merkle tree of the blockchain, judge whether the data in the storage node is modified. If it is modified, generate an incremental snapshot and add it to the Merkle tree, and a rollback operation can be performed based on the data acquisition time information; Protect the data security of each data in the storage node according to the dynamic desensitization technology and the national cipher SM4 algorithm.

2. The multi-source action data management method under the scalable distributed storage architecture according to claim 1, characterized in that The step of obtaining a multi-source data interface, performing an access operation on the original collected data information collected by the data collection module based on the multi-source data interface, and outputting a data access completion signal after the data access is completed specifically includes: Obtain a multi-source data interface and a data collection module, and establish a signal connection link between the multi-source data interface and the data collection module. Among them, the multi-source data interface is a unified access interface that supports multiple protocols; Collect information based on the data collection module to obtain original collected data information, and the original collected data information includes optical data information, IMU data information, and tactile data information; Perform an access operation on the original collected data information based on the multi-source data interface. After the access is completed, a data access completion signal is output. Among them, the multi-source data interface uses the NVMe protocol when accessing optical data information, the multi-source data interface uses the MQTT message queue when accessing IMU data information, and the multi-source data interface uses the gRPC stream method to access tactile data information.

3. The multi-source action data management method under the scalable distributed storage architecture according to claim 2, characterized in that, The step of, when the data access completion signal is received, performing a compression operation on the original collected data information to obtain multi-source compressed data information, performing timestamp calibration on the multi-source compressed data information, and marking the acquisition parameters of each data in the multi-source compressed data information specifically includes: After receiving the data access completion signal, compress the optical data information to obtain optical compressed data information, compress the IMU data information to obtain IMU compressed data information, and compress the tactile data information to obtain tactile compressed data information. The optical compressed data information, the IMU compressed data information, and the tactile compressed data information are combined to form multi-source compressed data information; Obtain the acquisition time of each data in the optical compressed data information, i.e., the optical data acquisition timestamp, obtain the acquisition time of each data in the IMU compressed data information, i.e., the IMU data acquisition timestamp, and obtain the acquisition time of each data in the tactile compressed data information, i.e., the tactile data acquisition timestamp; The optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp are combined to form data acquisition time information; According to the optical data acquisition timestamp, the IMU data acquisition timestamp, and the tactile data acquisition timestamp, perform timestamp calibration on the optical compressed data information, the IMU compressed data information, and the tactile data compressed information, and output a timestamp calibration completion signal after calibration; After receiving the timestamp calibration completion signal, judge and mark the data type information, device ID information, and spatial coordinate information of each data in the optical compressed data information, the IMU compressed data information, and the tactile compressed data information to obtain associated parameter marking information.

4. The multi-source action data management method under the scalable distributed storage architecture according to claim 3, characterized in that, The steps of dividing according to the data type and acquisition time of each data in the multi-source compressed data information to obtain data dynamic sharding strategy information, obtaining a storage cluster and judging the health of each storage node in the storage cluster, screening valid storage nodes, and dispersedly storing each data on the valid storage nodes based on the data dynamic sharding strategy information, and judging whether the storage cluster needs to be expanded. If expansion is required, perform an expansion operation based on a data migration algorithm specifically include: Preliminarily divide the multi-source compressed data information into multiple sub-sharded data according to the data type information to obtain data preliminary sharding information; According to the data acquisition time information, re-divide the data preliminary sharding information into multiple sub-sharded data to obtain data dynamic sharding strategy information; Obtain a storage cluster, which includes multiple storage nodes. Detect the health of each storage node. If the health of a storage node is low, mark the storage node as an invalid storage node, and mark the other storage nodes except the invalid storage nodes as valid storage nodes; Dispersedly store each sub-sharded data on each valid storage node based on the data dynamic sharding strategy information; According to the currently known storage capacity and the total storage capacity of the storage cluster, judge whether the storage cluster needs to add new storage nodes for expansion. If expansion is required, perform dynamic expansion based on the data migration algorithm of the consistent hashing ring; According to the access frequency and timeliness of each data in the multi-source compressed data information, perform cold and hot data stratification on each data in the multi-source compressed data information to obtain a cold and hot data division result. The cold and hot data division result includes cold data and hot data, and further compress the cold data based on erasure code technology.

5. The multi-source action data management method under the scalable distributed storage architecture according to claim 4, wherein The steps of obtaining and querying and reading each data in the storage cluster based on the user service layer specifically include: Obtain the user service layer, where the user service layer includes a query engine and a machine learning interface; Create a signal connection link between the query engine and the storage cluster. Based on the query engine, when a user inputs a query instruction, the query engine filters and displays the data in the storage cluster that meets the query instruction; Create a signal connection link between the machine learning interface and the storage cluster, and read each data in the storage cluster based on the machine learning interface.

6. The multi-source action data management method under the scalable distributed storage architecture according to claim 5, characterized in that The steps of constructing an R*-tree index structure for each data in the storage node and creating a metadata graph structure, where the R*-tree index structure and the metadata graph structure are combined to form a data location query structure, specifically include: Construct an R*-tree index structure for the spatial coordinate information in the data collection time information and the associated parameter marking information of each data in the storage node; Create a metadata graph structure according to the Neo4j graph database. The metadata graph structure includes multiple nodes and edges. The nodes include device ID nodes, data type nodes, and collection time nodes, and the edges represent the relationships between each data in the storage node; The R*-tree index structure and the metadata graph structure are combined to form a data location query structure.

7. The multi-source action data management method under the scalable distributed storage architecture according to claim 6, characterized in that The steps of uploading the data in the storage node to the Merkle tree of the blockchain, determining whether the data in the storage node is modified, generating an incremental snapshot and adding it to the Merkle tree if modified, and being able to perform a rollback operation based on the data collection time information specifically include: Upload the data in each storage node of the storage cluster to the Merkle tree of the blockchain. Among them, the leaf nodes of the Merkle tree store the hash values of each data, and the non-leaf nodes store the hash values of their child nodes; Determine whether the data in each storage node of the storage cluster is modified. If modified, generate an incremental snapshot after modification and add the incremental snapshot to the Merkle tree; If the user has data errors or needs to restore historical data, a rollback operation can be performed on the data based on the data collection time information.

8. The multi-source action data management method under the scalable distributed storage architecture according to claim 7, wherein The steps of protecting the data security of each data in the storage node according to the dynamic desensitization technology and the national secret SM4 algorithm specifically include: Perform real-time processing on each data in the storage node according to the dynamic desensitization technology, and add irreversible noise to each data in the storage node to meet K-anonymity ≥ 5; Obtain a hardware security module, and encrypt and store the cold data based on the national secret SM4 algorithm, where the key is entrusted by the hardware security module.

9. A multi-source action data management system under an extensible distributed storage architecture, characterized in that, The multi-source action data management system under the scalable distributed storage architecture is used to implement the multi-source action data management method under the scalable distributed storage architecture described in any one of claims 1-8, including: A data access module configured to obtain a multi-source data interface, perform an access operation on the original collection data information collected by the data collection module based on the multi-source data interface, and output a data access completion signal after the data access is completed; The data preprocessing module is configured to perform a compression operation on the original collected data information to obtain multi-source compressed data information after receiving the data access completion signal, perform timestamp calibration on the multi-source compressed data information, and mark the acquisition parameters of each data in the multi-source compressed data information; The storage design module is configured to divide according to the data type and acquisition time of each data in the multi-source compressed data information to obtain data dynamic sharding strategy information, obtain the storage cluster and judge the health of each storage node in the storage cluster, filter out valid storage nodes, and disperse each data to the valid storage nodes based on the data dynamic sharding strategy information, judge whether the storage cluster needs to be expanded, and if so, perform an expansion operation based on the data migration algorithm; The service layer acquisition module is configured to acquire and perform query operations and read operations on each data in the storage cluster based on the user service layer; The location query structure creation module is configured to construct an R* tree index structure and create a metadata graph structure based on each data in the storage node, and the R* tree index structure and the metadata graph structure are combined to form a data location query structure; The data traceability module is configured to upload the data in the storage node to the Merkle tree of the blockchain, judge whether the data in the storage node is modified, and if so, generate an incremental snapshot and add it to the Merkle tree, and a rollback operation can be performed based on the data acquisition time information; The data security management module is configured to protect the data security of each data in the storage node according to the dynamic desensitization technology and the national cryptography SM4 algorithm.

Images