Edge computing storage method, device and equipment based on multi-layer stacking structure

Abstract

The present invention relates to an edge computing storage method, device and equipment based on a multi-layer stacking structure, comprising the following steps: performing distributed feature extraction on the original data of a preset storage chip to obtain multi-dimensional data features; performing hierarchical and recursive segmentation on the original data based on the multi-dimensional data features to obtain multi-layer stacked data blocks; performing distributed index construction on the multi-layer stacked data blocks through a preset heterogeneous fusion algorithm to obtain a hierarchical index tree; performing topological compression mapping on the hierarchical index tree to obtain a compressed index structure; and allocating and storing the original data in a storage area corresponding to a target memory based on the compressed index structure, thereby solving the technical problem of how to effectively manage and utilize limited storage resources in a resource-constrained edge environment.

Description

Technical Field

[0001] The present invention relates to the field of chip technology, and in particular to an edge computing storage method, device and equipment based on a multi-layer stacking structure. Background Art

[0002] As an emerging computing model, edge computing aims to bring data processing and service provision as close as possible to data sources and end users, thereby reducing network latency, bandwidth consumption, and improving privacy protection. With the rapid development of Internet of Things (IoT) devices and applications, the importance of edge computing has become increasingly prominent. However, with the sharp increase in the number of smart devices connected to the Internet, the amount of data generated has also exploded, which has put tremendous pressure on storage systems. Traditional cloud-centric data processing methods are difficult to meet the requirements of real-time and efficiency, especially in resource-constrained edge environments. How to effectively manage and utilize limited storage resources has become an urgent problem to be solved.

[0003] To meet these challenges, researchers have proposed an edge computing storage method based on a multi-layer stacking structure. This method can more accurately understand the essential attributes of data through distributed feature extraction, thereby providing a basis for subsequent hierarchical and recursive segmentation of data. It can not only improve the efficiency of data organization, but also optimize the access path of data. However, current storage technology still faces problems such as high data redundancy and low query efficiency when processing large-scale data. In addition, building an effective indexing mechanism to support fast retrieval is also one of the difficulties, because existing indexing technologies may not be well adapted to dynamic data changes and heterogeneous hardware architectures in edge environments.

[0004] In response to the above problems, this study proposed an innovative method to improve data storage and management in edge computing. By extracting multi-dimensional features from the original data and using hierarchical recursive segmentation to form multi-layer stacked data blocks, this method attempts to achieve more fine-grained data segmentation to better match the needs of different application scenarios. Subsequently, a hierarchical index tree is constructed using a preset heterogeneous fusion algorithm, and topological compression mapping is performed on it to create an efficient compressed index structure. The ultimate goal is to ensure that the original data can be reasonably allocated and efficiently stored in the target memory, thereby improving the performance of the entire system while reducing energy consumption and maintenance costs. This method is expected to bring new breakthroughs in the field of edge computing, especially in promoting the development of large-scale projects such as smart cities and Industry 4.0. Summary of the invention

[0005] The main purpose of the present invention is to provide an edge computing storage method, device and equipment based on a multi-layer stacking structure, which solves the technical problem of how to effectively manage and utilize limited storage resources in a resource-constrained edge environment.

[0006] To achieve the above object, the present invention provides an edge computing storage method based on a multi-layer stacking structure, which is applied to a storage chip and includes the following steps:

[0007] Perform distributed feature extraction on the original data of the preset storage chip to obtain multi-dimensional data features;

[0008] Based on the multi-dimensional data features, the original data is recursively divided into layers to obtain multi-layer stacked data blocks;

[0009] Performing distributed index construction on the multi-layer stacked data blocks through a preset heterogeneous fusion algorithm to obtain a hierarchical index tree;

[0010] Performing topological compression mapping on the hierarchical index tree to obtain a compressed index structure;

[0011] The original data is allocated and stored in a storage area corresponding to the target memory based on the compressed index structure.

[0012] Furthermore, the distributed feature extraction is performed on the original data of the preset storage chip to obtain multi-dimensional data features, including:

[0013] Using the distributed computing framework of the storage chip, the original data of the preset storage chip is initially decomposed to obtain edge feature data; wherein the original data comes from the original data collected by different types of sensors;

[0014] Slice the edge feature data to obtain a data slice set;

[0015] Performing feature dimensionality reduction on the data shard set by using a distributed local sensitive hashing algorithm to obtain a reduced dimensionality data shard set;

[0016] The cross-sensor feature fusion is performed on the dimension-reduced data slice set through a preset feature fusion algorithm to obtain multi-dimensional data features.

[0017] Furthermore, the original data is hierarchically and recursively segmented based on the multi-dimensional data features to obtain multi-layer stacked data blocks, including:

[0018] Adaptively quantize and segment the multidimensional data features to obtain a feature segmentation sequence, and perform nonlinear boundary detection on the feature segmentation sequence to obtain a hierarchical boundary set;

[0019] Based on the hierarchical boundary set, the original data is hierarchically and recursively segmented to obtain an initial data block set, and a tensor decomposition operation is performed on the initial data block set to obtain a multi-dimensional tensor block sequence;

[0020] Performing spatiotemporal correlation analysis on the multidimensional tensor block sequence through a recursive neural network to obtain a correlation feature map between the original data, and performing hierarchical clustering processing on the correlation feature map to obtain a multi-level data block group;

[0021] Reconstructing the topological structure of the multi-level data block group to obtain a hierarchical data block network, and dynamically balancing and optimizing the hierarchical data block network to obtain a balanced data block sequence;

[0022] Performing inter-block relationship reasoning on the balanced data block sequence through a graph neural network to obtain a data block association graph, and performing edge weight calculation on the data block association graph to obtain a weighted data block set;

[0023] The original data is reorganized into a stacked structure based on the weighted data block set to obtain multi-layer stacked data blocks.

[0024] Furthermore, the distributed index construction is performed on the multi-layer stacked data blocks by using a preset heterogeneous fusion algorithm to obtain a hierarchical index tree, including:

[0025] Performing heterogeneous feature extraction on the multi-layer stacked data blocks to obtain multimodal feature vectors, and performing cross-modal similarity calculation on the multimodal feature vectors to obtain heterogeneous data association data;

[0026] Performing local sensitive hash coding on the heterogeneous data association data to obtain a distributed hash index table;

[0027] Using a distributed consistent hashing algorithm to perform load balancing mapping on the distributed hash index table to obtain a balanced index shard set, and constructing a Bloom filter on the balanced index shard set to obtain a fast query index;

[0028] Constructing a multi-level index hierarchy based on the fast query index, and performing B+ tree conversion on the multi-level index hierarchy to obtain a hybrid index structure;

[0029] Distributed segmentation of the hybrid index structure is performed using a graph partitioning algorithm to obtain a subgraph index set, and a minimum spanning tree is constructed for the subgraph index set to obtain an index tree skeleton;

[0030] The index tree skeleton is subjected to index tree pruning and hierarchical index optimization to obtain a hierarchical index tree, wherein the hierarchical index tree includes node identifiers, hierarchical relationships, data block locations, and index mapping relationships.

[0031] Furthermore, the topological compression mapping of the hierarchical index tree to obtain a compressed index structure includes:

[0032] Performing graph embedding representation on the hierarchical index tree to obtain an index graph embedding vector; wherein the index graph embedding vector includes a node vector and an edge vector, the node vector dimension is 128, and the edge vector dimension is 64;

[0033] Constructing a topological skeleton graph based on the index graph embedding vector; wherein the topological skeleton graph comprises skeleton nodes and skeleton edges, the skeleton nodes represent key index nodes, and the skeleton edges represent key index paths;

[0034] Extracting topological features from the topological skeleton map using a Persistent Homology algorithm to obtain a persistent barcode; wherein the persistent barcode represents the duration of topological features at different scales;

[0035] Constructing a topological fingerprint vector based on the persistent barcode, and performing a nonlinear transformation on the topological fingerprint vector to obtain a compressed topological feature vector; wherein the topological fingerprint vector dimension is 256 and is constructed using a MinHash algorithm;

[0036] The compressed topological feature vector is indexed and encoded by a local sensitive hashing algorithm to obtain a compressed index structure; wherein the local sensitive hashing algorithm adopts the SimHash algorithm, and the hash code length is 128 bits.

[0037] Further, allocating and storing the original data in a storage area corresponding to the target memory based on the compressed index structure includes:

[0038] Obtain a distributed storage resource map of the current storage chip, perform dynamic resource scheduling on the original data based on the distributed storage resource map to obtain a storage resource scheduling strategy, and perform topology-aware mapping on the storage resource scheduling strategy to obtain a storage location mapping table,

[0039] Distributed data sharding is established based on the storage location mapping table to obtain a data sharding scheme, and an optimal transmission path is calculated for the original data based on the data sharding scheme to obtain a data transmission topology diagram;

[0040] Perform elastic scaling control on the data transmission topology diagram to obtain a dynamic allocation solution;

[0041] The original data is allocated and stored in a storage area corresponding to the target memory based on the dynamic allocation scheme.

[0042] Furthermore, the step of calculating the optimal transmission path for the original data based on the data sharding scheme to obtain a data transmission topology diagram includes:

[0043] Performing a topological complexity analysis on the data sharding scheme to obtain sharding transmission constraints, and performing a multi-objective optimization solution on the sharding transmission constraints to obtain transmission cost data, wherein the transmission cost data includes bandwidth consumption, delay overhead, and energy loss;

[0044] Performing distributed routing planning on the original data based on the transmission cost data to obtain a candidate path set, and performing annealing search on the candidate path set to obtain a local optimal path sequence;

[0045] Performing Pareto front analysis on the local optimal path sequence by a non-dominated sorting algorithm to obtain a multi-criteria evaluation index, and performing fuzzy hierarchical analysis on the multi-criteria evaluation index to obtain a path priority table;

[0046] Constructing a transmission topology based on the path priority table to obtain an initial transmission network, and performing hypergraph segmentation processing on the initial transmission network to obtain subnetwork clusters;

[0047] Performing collaborative game optimization on the sub-network cluster to obtain a Nash equilibrium solution set, and performing topological entropy calculation on the Nash equilibrium solution set to obtain an entropy value sequence;

[0048] The sub-network cluster is reconstructed for topological transmission based on the entropy value sequence to obtain a data transmission topology graph.

[0049] The present invention also provides an edge computing storage device based on a multi-layer stacking structure, which is applied to a storage chip, including:

[0050] An extraction module is used to perform distributed feature extraction on the raw data of the preset storage chip to obtain multi-dimensional data features;

[0051] A segmentation module, used for performing hierarchical and recursive segmentation on the original data based on the multi-dimensional data features to obtain multi-layer stacked data blocks;

[0052] A construction module, used to perform distributed index construction on the multi-layer stacked data blocks through a preset heterogeneous fusion algorithm to obtain a hierarchical index tree;

[0053] A mapping module, used for performing topological compression mapping on the hierarchical index tree to obtain a compressed index structure;

[0054] An allocation module is used to allocate and store the original data in a storage area corresponding to the target memory based on the compressed index structure.

[0055] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0056] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0057] The edge computing storage method based on a multi-layer stacking structure provided by the present invention includes the following steps: performing distributed feature extraction on the original data of a preset storage chip to obtain multi-dimensional data features; performing hierarchical recursive segmentation on the original data based on the multi-dimensional data features to obtain multi-layer stacking data blocks; performing distributed index construction on the multi-layer stacking data blocks through a preset heterogeneous fusion algorithm to obtain a hierarchical index tree; performing topological compression mapping on the hierarchical index tree to obtain a compressed index structure; allocating and storing the original data in the storage area corresponding to the target memory based on the compressed index structure, through the above-mentioned technical means, the technical problem of how to effectively manage and utilize limited storage resources in a resource-constrained edge environment is solved, and the hierarchical index tree is formed into a compressed index structure after topological compression mapping, which greatly simplifies the index system and reduces the number of hierarchical traversals required for querying. This means that when specific data needs to be accessed, the system can locate the target data block more quickly, improving the technical effect of data retrieval speed and response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the steps of an edge computing storage method based on a multi-layer stacking structure in one embodiment of the present invention;

[0059] Figure 2 is a structural block diagram of an edge computing storage device based on a multi-layer stacking structure in one embodiment of the present invention;

[0060] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0061] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] like Figure 1 As shown, Figure 1 This is a schematic diagram of the steps of an edge computing storage method based on a multi-layer stacking structure in one embodiment of the present invention;

[0064] In one embodiment of the present invention, an edge computing storage method based on a multi-layer stacking structure is provided, which is applied to a storage chip and includes the following steps:

[0065] Step S1, performing distributed feature extraction on the original data of the preset storage chip to obtain multi-dimensional data features.

[0066] Specifically, when implementing the step of performing distributed feature extraction on the raw data of the preset storage chip to obtain multi-dimensional data features, it is first necessary to understand that in the edge computing environment, the data sources are wide and diverse, and these data may come from various sensors, surveillance cameras or other IoT devices. In order to make these raw data more efficiently used in the subsequent processing process, they must first be feature extracted. The "distributed feature extraction" mentioned here refers to running specific algorithms on multiple nodes deployed at the edge of the network, and processing data from different sources in parallel, rather than uploading all data to the cloud for unified analysis. For example, in an intelligent transportation system, cameras distributed throughout the city will continuously generate video stream data. If you want to monitor traffic flow conditions in real time and warn of abnormal conditions, each camera node can be used as a distributed feature extraction point. After receiving the original video frame, the local processor quickly extracts key information from each frame of the image according to preset rules, such as color histograms, texture features or motion vectors, thereby forming a multi-dimensional data feature set. This process not only reduces the pressure on network transmission, but also greatly shortens the time delay from data acquisition to actual application. Next, due to the high-dimensional and unstructured characteristics of data in edge computing environments, direct analysis of them may encounter the problem of dimensionality disaster, that is, as the number of features increases, the computational complexity increases exponentially. Therefore, the "multidimensional data features" mentioned here mean a set of numerical or symbolic representations that can accurately describe the core attributes of the original data after preliminary screening and conversion. Continuing with the example of intelligent transportation, by quantifying multiple dimensions such as the color, shape, size of the video frame, and the speed and direction of the vehicle, a compact and expressive data feature vector can be constructed. Such a feature vector not only retains the important information of the original data, but also facilitates subsequent hierarchical recursive segmentation and other advanced processing tasks, laying the foundation for improving the response speed and accuracy of the entire system. In this process, maintaining data integrity and privacy is also one of the most important considerations to ensure that feature extraction can be completed safely and reliably even in a distributed environment.

[0067] Step S2, based on the multi-dimensional data features, the original data is recursively divided into layers to obtain multi-layer stacked data blocks.

[0068] Specifically, the process of hierarchical recursive segmentation of the raw data based on the multidimensional data features to obtain multi-layer stacked data blocks is a key step in the edge computing storage method. In the intelligent transportation system, this process can significantly improve the efficiency and accuracy of data analysis. Specifically, after completing the distributed feature extraction, the system has obtained multi-dimensional data features about each frame of the video stream, which may include information such as the color, shape, size, and movement direction of the vehicle. Next, in order to further optimize the data management and processing process, the system needs to organize the raw data according to these multi-dimensional data features. The "hierarchical recursive segmentation" mentioned here refers to the step-by-step decomposition of the raw data into smaller data units according to certain logical rules, that is, the formation of so-called "multi-layer stacked data blocks". For example, in the intelligent traffic monitoring scenario, all collected data can be first divided according to geographic location information (such as the intersection where the camera is located), and then the data in each geographic area can be further subdivided according to other dimensions such as timestamps or vehicle types. Such recursive segmentation not only helps to establish a clear data hierarchy, but also enables data at different levels to be called and analyzed independently or in combination according to actual needs. In addition, "multi-layer stacking" means that there is a clear hierarchical relationship between the data blocks constructed in this hierarchical manner. The lower-level data blocks may be further refined from the higher-level data blocks, while the higher-level data blocks summarize the information of multiple data blocks in the next level. Taking intelligent transportation as an example, the highest-level data blocks may contain an overview of traffic flow in the entire city, the middle-level data blocks represent the traffic conditions in a specific area, and the lowest-level data blocks are detailed records of a specific intersection at a certain moment. This structured data organization form provides a solid foundation for subsequent index construction and is essential for achieving fast data retrieval and efficient space utilization. When a user wants to query the traffic conditions of a certain intersection within a specific time period, the system can quickly locate the required information by traversing the corresponding multi-layer stacked data blocks, thereby greatly improving the response speed and service quality. At the same time, this method also helps to reduce data redundancy, reduce unnecessary storage overhead, and ensure the effective use of resources.

[0069] Step S3, constructing a distributed index for the multi-layer stacked data blocks by using a preset heterogeneous fusion algorithm to obtain a hierarchical index tree.

[0070] Specifically, after realizing the hierarchical recursive segmentation of multi-layer stacked data blocks, the next step is to construct a distributed index for these data blocks through a preset heterogeneous fusion algorithm to obtain a hierarchical index tree. This process is crucial to improving data retrieval efficiency and optimizing storage structure. In intelligent transportation systems, as the camera network continues to expand, the amount of video data is growing exponentially, and how to quickly locate and obtain specific information has become a challenge. To address this problem, the system needs an effective method to organize and manage these massive amounts of data. In this case, it is particularly critical to use heterogeneous fusion algorithms to build indexes. Heterogeneous fusion refers to integrating data from different sources, formats, or characteristics into a unified framework and leveraging their respective advantages to improve overall performance. For example, in intelligent traffic monitoring applications, we can combine time series analysis, geolocation encoding, and vehicle feature recognition to create corresponding index nodes for each layer of stacked data blocks. These nodes not only contain basic descriptive information about the data layer, such as time period, geographic area, etc., but also record deeper data features, such as vehicle type, speed mode, etc. In this way, when the user wants to query the traffic conditions of a certain intersection within a certain period of time, the system can quickly locate the relevant index nodes and search downward step by step through the hierarchical relationship until the bottom-level specific data block is found, which greatly improves the retrieval efficiency. At the same time, since the index is constructed in a distributed manner, that is, different data blocks are processed in parallel on multiple computing nodes, this can not only speed up the speed of index generation, but also enhance the fault tolerance and scalability of the system. Even if some nodes fail, other nodes can continue to work to ensure the stable operation of the entire system. In addition, this distributed architecture can also adapt to the uneven distribution of resources in the edge computing environment, make full use of the computing power and storage space of each node, and achieve the optimal configuration in the global scope. The resulting hierarchical index tree not only provides a clear logical structure for the data, but also facilitates subsequent maintenance and update operations. Whenever new data is added or old data is modified, only the corresponding index node needs to be adjusted without rebuilding the entire index system, which further improves the flexibility and response speed of the system. In summary, by using a preset heterogeneous fusion algorithm to construct a distributed index for multi-layer stacked data blocks, we can effectively solve the massive data management and rapid retrieval problems faced by intelligent transportation systems, and provide users with a more convenient and efficient service experience. At the same time, this method also provides a feasible technical path for big data processing in other similar application scenarios, and promotes the development and application of edge computing technology.

[0071] Step S4, performing topological compression mapping on the hierarchical index tree to obtain a compressed index structure.

[0072] Specifically, after the hierarchical index tree is constructed, the next step is to perform topological compression mapping on the hierarchical index tree to obtain a compressed index structure. This process aims to optimize the space occupancy and query efficiency of the index tree to ensure that it can run efficiently in a resource-constrained edge computing environment. In intelligent transportation systems, as the amount of data continues to grow, the original hierarchical index tree may become too large and complex, resulting in increased storage costs and longer query response times. Therefore, it is particularly important to streamline and optimize the index tree through topological compression mapping technology. Specifically, topological compression mapping refers to the process of removing redundant information and simplifying the hierarchical structure based on the relationship between nodes within the index tree and the data features they represent. In the intelligent traffic monitoring scenario, it is assumed that we have built a detailed hierarchical index tree based on factors such as geographic location, timestamps, and vehicle characteristics. This tree may contain multiple layers of index nodes from the city level to the specific intersection level, and each node records the traffic flow profile in the corresponding area. However, not all nodes are necessary for every query. For example, when retrieving detailed traffic conditions at a certain intersection at a certain time, users usually do not need to care about the macro traffic trends of the entire city, and the high-level node information can be appropriately omitted. By merging or deleting these unnecessary nodes, the overall size of the index tree can be significantly reduced while retaining important information on the critical path. This compression not only saves storage space, but also makes the index structure more compact and easy to traverse quickly. More importantly, the compressed index structure can better adapt to the limited computing power and memory resources of edge devices. For example, in a distributed camera network, the processing unit on each node can load the compressed index structure, thereby achieving localized fast query and analysis without relying on the support of remote servers. This not only improves the response speed of the system, but also reduces the demand for network bandwidth. In addition, topological compression mapping also involves the redesign of the topological structure of the index tree to ensure that the original logical relationship and query accuracy can be maintained even after compression. In this process, the algorithm will take into account the importance differences between different nodes and give priority to retaining those nodes that have a greater impact on the query results. For example, in an intelligent transportation system, information nodes involving accident-prone areas or traffic-controlled sections will be given higher weights to ensure that they will not be lost due to compression. The resulting compressed index structure is not only compact in size, but also has excellent query performance, providing users with an efficient and reliable data access platform, and also provides a useful technical reference for other similar application scenarios.

[0073] Step S5: allocating and storing the original data in a storage area corresponding to the target memory based on the compressed index structure.

[0074] Specifically, allocating and storing the original data in the storage area corresponding to the target memory based on the compressed index structure is the last important step in the edge computing storage method. This process ensures that the data after feature extraction, hierarchical recursive segmentation, index tree construction and compression can be reasonably organized and efficiently stored in the appropriate physical location. In the intelligent transportation system, this means that after a series of processing, the video stream data can be accurately mapped to the most suitable storage unit according to its multidimensional features and spatiotemporal attributes, thereby optimizing resource utilization and improving retrieval efficiency. To achieve this, it is first necessary to determine the optimal storage solution for each layer of stacked data blocks based on the compressed index structure. Since the compressed index structure has simplified the hierarchical relationship and removed redundant information through topological compression mapping, it provides clear guidance for the final placement of the data. For example, in an intelligent transportation monitoring network, for a video clip of a certain intersection within a certain period of time, if the section is located in the core area of ​​the city and has a large traffic volume, these data may be given priority to be stored on a fast access medium close to the edge node, such as a solid state drive (SSD), to facilitate real-time analysis and rapid response to query requests. On the contrary, relatively inactive or older data can be arranged in traditional hard disks (HDDs) or other types of long-term storage devices with lower costs but larger capacity. At the same time, considering the performance differences between different storage devices, the allocation strategy should also take into account factors such as read and write speed and access frequency. For example, in a distributed camera network, the central server may be equipped with a high-performance storage array to store relevant data of key intersections or important events, while the edge nodes distributed in various places are configured with an appropriate amount of local cache space to temporarily store temporarily generated data. When new data arrives, the system will automatically determine where it should be stored according to the preset rules, and quickly locate the corresponding target storage area through the compressed index structure. In this way, not only the timeliness and availability of data are guaranteed, but also the load balancing of the entire system is achieved. In addition, in order to further improve storage efficiency, intelligent scheduling algorithms can be introduced to dynamically adjust the layout of data. For example, when facing a sudden influx of large-volume data, the system can flexibly select idle resources for expanded storage according to the current working status of each storage node; in daily operations, data migration operations are performed regularly to gradually transfer cold data that is no longer frequently used to more economical storage media. This intelligent allocation mechanism based on compressed index structure not only meets the needs of intelligent transportation system for massive data management, but also effectively reduces operation and maintenance costs and improves the overall performance and service quality of the system. In this way, even in complex and changing application environments, it can ensure that each piece of data can find the most suitable "home" for itself, providing users with a more stable and reliable service experience.

[0075] In a specific embodiment, the distributed feature extraction is performed on the raw data of the preset storage chip to obtain multi-dimensional data features, including:

[0076] Using the distributed computing framework of the storage chip, the original data of the preset storage chip is initially decomposed to obtain edge feature data; wherein the original data comes from the original data collected by different types of sensors;

[0077] Slice the edge feature data to obtain a data slice set;

[0078] Performing feature dimensionality reduction on the data shard set by using a distributed local sensitive hashing algorithm to obtain a reduced dimensionality data shard set;

[0079] The cross-sensor feature fusion is performed on the dimension-reduced data slice set through a preset feature fusion algorithm to obtain multi-dimensional data features.

[0080] Specifically, the process of extracting distributed features from the raw data of the preset storage chip to obtain multidimensional data features is an important application example of the combination of edge computing and big data processing technology. In this process, the distributed computing framework of the storage chip is first used to process the raw data collected from different types of sensors to achieve initial feature decomposition, thereby obtaining edge feature data. The "edge" here refers to the place close to the data source, that is, the location where the data is generated and initially processed, which can reduce transmission delays and improve response speed. For example, in an intelligent transportation system, cameras, temperature sensors, humidity sensors and other devices installed at various intersections will continuously generate a large amount of raw data such as images, video streams and environmental parameters. These data are analyzed instantly by local edge computing nodes to identify basic features such as vehicle type, license plate number, road conditions, etc., which constitute edge feature data. Next, in order to further optimize data processing efficiency and adapt to the requirements of distributed environments, the system will shard the above edge feature data to obtain a data shard set. This sharding strategy not only helps to disperse the computing load, but also ensures that each data shard remains relatively independent logically, which is convenient for parallel processing. Continuing with the example of intelligent transportation, data from the same camera or cameras in adjacent areas may be divided into multiple small batches, each of which contains a snapshot of traffic conditions within a specific time period. This can reduce the pressure on a single node and improve the response speed of the overall system, so that even in a busy urban traffic monitoring network, it can quickly respond to the situation in the local area. Then, the system uses a distributed local sensitive hashing algorithm (LSH) to reduce the feature dimension of the data shard set to obtain a reduced dimension data shard set. Local sensitive hashing is an efficient approximate nearest neighbor search technology that can significantly reduce the spatial complexity of high-dimensional data while ensuring a certain accuracy. In the intelligent transportation application scenario, this means that even in the face of a large number of similar but slightly different vehicle images, the system can quickly find the most relevant samples and map them to a lower-dimensional space representation. This not only speeds up data retrieval, but also reduces storage requirements, which is particularly important for large-scale monitoring systems. Finally, the reduced dimension data shard set is cross-sensor feature fusion through a preset feature fusion algorithm to obtain multi-dimensional data features. This step is to integrate data shards from different sources and build a more comprehensive and abstract data representation. Continuing with the example of the smart traffic scenario, different types of sensors (such as visual information captured by cameras and ambient temperature recorded by temperature sensors) generate different angle views of the same car. After dimensionality reduction processing, they are combined by a feature fusion algorithm to form a multi-dimensional data feature that contains all-round information about the vehicle.This not only enhances the depth of understanding of a single target object, but also provides a solid foundation for subsequent advanced analysis tasks such as path prediction and behavior pattern recognition. In summary, the entire distributed feature extraction process not only retains the key characteristics of the original data, but also achieves efficient data organization and management, providing a strong technical guarantee for real-time decision support for intelligent transportation systems. In this way, even when faced with massive and diverse sensor data, the system can still quickly and accurately extract useful information to help managers better understand and respond to complex traffic conditions. This method is not only applicable to the field of intelligent transportation, but can also be extended to other application scenarios that require processing of large amounts of heterogeneous data, such as smart cities, industrial Internet of Things, etc. As long as it involves the need to obtain and effectively utilize data from multiple sources, this solution can play its unique advantages.

[0081] In a specific embodiment, the layered and recursive segmentation of the original data based on the multi-dimensional data features to obtain multi-layer stacked data blocks includes:

[0082] Adaptively quantize and segment the multidimensional data features to obtain a feature segmentation sequence, and perform nonlinear boundary detection on the feature segmentation sequence to obtain a hierarchical boundary set;

[0083] Based on the hierarchical boundary set, the original data is hierarchically and recursively segmented to obtain an initial data block set, and a tensor decomposition operation is performed on the initial data block set to obtain a multi-dimensional tensor block sequence;

[0084] Performing spatiotemporal correlation analysis on the multidimensional tensor block sequence through a recursive neural network to obtain a correlation feature map between the original data, and performing hierarchical clustering processing on the correlation feature map to obtain a multi-level data block group;

[0085] Reconstructing the topological structure of the multi-level data block group to obtain a hierarchical data block network, and dynamically balancing and optimizing the hierarchical data block network to obtain a balanced data block sequence;

[0086] Performing inter-block relationship reasoning on the balanced data block sequence through a graph neural network to obtain a data block association graph, and performing edge weight calculation on the data block association graph to obtain a weighted data block set;

[0087] The original data is reorganized into a stacked structure based on the weighted data block set to obtain multi-layer stacked data blocks.

[0088] Specifically, the process of hierarchically and recursively segmenting the original data based on the multidimensional data features to obtain multi-layer stacked data blocks is achieved through a series of complex and sophisticated steps. First, the system will adaptively and quantitatively segment the multidimensional data features to obtain a feature segmentation sequence. This operation is intended to dynamically adjust the segmentation granularity according to the distribution characteristics of the data itself to ensure that each segmented subset can better reflect a part of the attributes of the original data. Then, in order to further refine these segmentation results, the system will perform nonlinear boundary detection on the feature segmentation sequence to obtain a hierarchical boundary set. In the intelligent traffic monitoring scenario, for example, the system can preliminarily classify objects in the video frame according to features such as vehicle speed, color or vehicle model, and identify the natural boundaries between different categories, providing a clear structural framework for subsequent data processing. Then, based on the above hierarchical boundary set, the system will start the recursive block segmentation process of the original data to generate an initial data block set. This block division method is not a simple uniform division, but takes into account the logical association and spatial distribution characteristics within the data, so that each data block can contain sufficient but not redundant information. Afterwards, by performing tensor decomposition operations on the initial data block set, the system can obtain a multi-dimensional tensor block sequence. In this process, the video data of the intelligent transportation system is converted into a higher-dimensional mathematical representation, which not only helps to capture deeper data features, but also facilitates subsequent complex analysis and calculation. For example, through tensor decomposition, the system can comprehensively understand the content of a video from multiple perspectives (time, location, object features, etc.), rather than looking at each frame of the image in isolation. Next, a recurrent neural network (RNN) is used to perform spatiotemporal correlation analysis on the multi-dimensional tensor block sequence to generate a correlation feature map between the original data. RNN is good at processing data with time series properties, so it is very suitable for revealing the pattern of traffic flow changes over time and space. For example, when analyzing urban road networks, RNN can help identify which sections of roads are often congested during a specific time period, and how these congestions affect the traffic conditions in the surrounding areas. Subsequently, the system will perform hierarchical clustering on the correlation feature map to form a multi-level data block group. This means that similar data blocks will be classified into the same level, while data blocks of different levels represent different levels of abstraction, gradually building an orderly data organization system from concrete to abstract. As multi-level data block groups are established, the system needs to reconstruct their topology to create a hierarchical data block network. This step is to better reflect the connection relationship between data blocks and ensure that they can be stored together logically coherently and physically compactly. In addition, in order to optimize resource allocation and improve access efficiency, the system will also dynamically balance and optimize the hierarchical data block network to obtain a balanced data block sequence. In an intelligent transportation environment, this means that even if the amount of data on certain sections of the road suddenly increases, the system can flexibly adjust the storage layout to ensure that the overall performance is not affected.For example, when a traffic accident occurs at a certain intersection and a large amount of video data flows in, the system can quickly find a suitable storage location through a dynamic balancing mechanism to avoid local overload. Finally, by using the graph neural network (GNN) to reason about the inter-block relationship of the balanced data block sequence, the system can generate a data block association graph and calculate the edge weights in the association graph to finally obtain a weighted data block set. GNN is good at processing data with complex relationship structures. It can assign different weight values ​​according to the strength of association between each data block, so as to more accurately describe the interaction between them. In intelligent transportation applications, such a weighted data block set can provide strong support for traffic management decisions, help predict potential risk points or optimize traffic light control strategies. Based on the weighted data block set, the system can stack the original data and reorganize it to form a multi-layer stacked data block. This structured data organization method not only improves the speed of data retrieval, but also facilitates future data analysis and mining. In summary, through this series of carefully designed steps, the system realizes efficient management and optimized storage of raw data, ensuring that the intelligent traffic monitoring system can process massive data in real time while quickly responding to user query needs and providing accurate and reliable traffic information.

[0089] In a specific embodiment, the distributed index construction is performed on the multi-layer stacked data blocks by a preset heterogeneous fusion algorithm to obtain a hierarchical index tree, including:

[0090] Performing heterogeneous feature extraction on the multi-layer stacked data blocks to obtain multimodal feature vectors, and performing cross-modal similarity calculation on the multimodal feature vectors to obtain heterogeneous data association data;

[0091] Performing local sensitive hash coding on the heterogeneous data association data to obtain a distributed hash index table;

[0092] Using a distributed consistent hashing algorithm to perform load balancing mapping on the distributed hash index table to obtain a balanced index shard set, and constructing a Bloom filter on the balanced index shard set to obtain a fast query index;

[0093] Constructing a multi-level index hierarchy based on the fast query index, and performing B+ tree conversion on the multi-level index hierarchy to obtain a hybrid index structure;

[0094] Distributed segmentation of the hybrid index structure is performed using a graph partitioning algorithm to obtain a subgraph index set, and a minimum spanning tree is constructed for the subgraph index set to obtain an index tree skeleton;

[0095] The index tree skeleton is subjected to index tree pruning and hierarchical index optimization to obtain a hierarchical index tree, wherein the hierarchical index tree includes node identifiers, hierarchical relationships, data block locations, and index mapping relationships.

[0096] Specifically, in the process of constructing a hierarchical index tree, the first thing involved is to process multi-layer stacked data blocks. The multi-layer stacked data blocks here refer to data sets of different sources, formats or properties that are combined to form a hierarchical data body. For example, in a large multimedia search engine, these data may include text, images, video clips, etc., which are organized into multiple levels to reflect their internal relationships and connections. In order to effectively manage and retrieve such heterogeneous data, it is necessary to create an index system that can be quickly located and accessed through a preset heterogeneous fusion algorithm. When starting to process these multi-layer stacked data blocks, the first step is to perform heterogeneous feature extraction. This step aims to extract meaningful feature vectors from different data types, namely the so-called multimodal feature vectors. The features of each data type may be different. For example, text data can obtain features through word frequency statistics, topic models, etc.; image data can obtain feature descriptions using methods such as color histograms and texture analysis. Video data may combine audio features with visual features to form multimodal feature vectors. Then, for these multimodal feature vectors, it is necessary to calculate the cross-modal similarity between them to establish heterogeneous data association data. This means finding potential correlations or similarities between different forms of data so that they can be better linked in subsequent operations. For example, in a database containing a large number of book cover images and corresponding book introduction texts, we hope to accurately match the correct book record even if there are only images or only text information. Next, for the obtained heterogeneous data association data, the locality sensitive hashing (LSH) encoding technology is applied to generate a distributed hash index table. LSH is a special hash function design that makes similar objects more likely to be mapped to the same bucket, which can greatly improve the speed of approximate nearest neighbor search. At this stage, our goal is to create a distributed index structure that can support efficient queries, so it is necessary to ensure that good performance can be maintained even in a distributed environment. Specifically, it is to assign a large amount of heterogeneous data association data to different nodes and ensure that the data on each node can be quickly accessed. The result of this step is a distributed hash index table, which not only contains the location information of the original data, but also implies the similarity between each data point. With the establishment of the distributed hash index table, the distributed consistent hashing algorithm is used to load balance it, and then a balanced index shard set is obtained. The consistent hashing algorithm ensures that even when the network topology changes (such as nodes joining or leaving), only minimal data migration occurs while maintaining a relatively uniform data distribution. This is critical for the stability of large distributed systems. On this basis, Bloom filters are further implemented on the balanced index shard collection to obtain fast query indexes.Bloom filter is a probabilistic data structure that can be used to determine whether an element exists in a set. Although there may be false positives, there will never be false negatives. Using Bloom filter can help us exclude those query requests that are definitely not in the target set in the initial screening stage, greatly reducing unnecessary I / O operations. Based on the fast query index, we can now start to build a multi-level index hierarchy. This process involves reorganizing the existing index structure to make it more suitable for specific application requirements. For example, the index can be divided into different levels according to factors such as data importance and access frequency. After that, these multi-level index hierarchies need to be converted into B+ trees to form a hybrid index structure. B+ tree is a balanced search tree that is particularly suitable for disk storage. It allows fast insertion, deletion and search operations, and all leaf nodes are on the same level, which is conducive to range queries. In our multimedia search engine example, such a hybrid index structure helps to improve query efficiency, especially when facing massive data. Finally, in order for the entire index system to work properly in a distributed environment, the hybrid index structure must be distributed and split through the graph partitioning algorithm to generate a subgraph index set. This is done to ensure that each subgraph can independently process a part of the query request, and it is also convenient for parallel processing. Next, we need to construct a minimum spanning tree for these subgraph index sets to obtain the index tree skeleton. The minimum spanning tree is a connected, acyclic subgraph with the smallest sum of edge weights. It provides a basic framework for the index tree and ensures the connectivity between the various parts. Once we have this index tree skeleton, we can continue to perform index tree pruning and hierarchical index optimization to finally obtain a hierarchical index tree. This final form of the index tree not only contains node identification, hierarchical relationships, data block locations, and index mapping relationships, but also has efficient query capabilities and good scalability, which is sufficient to cope with complex and changing data environments. In summary, starting from heterogeneous feature extraction of multi-layer stacked data blocks to completing the construction of a hierarchical index tree, this series of steps constitutes a complete process designed to ensure the efficiency and accuracy of the index system. The work at each stage is to preserve the important information of the original data as much as possible while minimizing complexity and resource consumption. This method can not only be applied to application scenarios such as multimedia search engines, but can also be extended to other fields involving large-scale heterogeneous data management, such as social network analysis, bioinformatics research, etc. As long as it involves complex hierarchical relationships and efficient retrieval needs, this solution can play its unique advantages.

[0097] In a specific embodiment, the topological compression mapping of the hierarchical index tree to obtain a compressed index structure includes:

[0098] Performing graph embedding representation on the hierarchical index tree to obtain an index graph embedding vector; wherein the index graph embedding vector includes a node vector and an edge vector, the node vector dimension is 128, and the edge vector dimension is 64;

[0099] Constructing a topological skeleton graph based on the index graph embedding vector; wherein the topological skeleton graph comprises skeleton nodes and skeleton edges, the skeleton nodes represent key index nodes, and the skeleton edges represent key index paths;

[0100] Extracting topological features from the topological skeleton map using a Persistent Homology algorithm to obtain a persistent barcode; wherein the persistent barcode represents the duration of topological features at different scales;

[0101] Constructing a topological fingerprint vector based on the persistent barcode, and performing a nonlinear transformation on the topological fingerprint vector to obtain a compressed topological feature vector; wherein the topological fingerprint vector dimension is 256 and is constructed using a MinHash algorithm;

[0102] The compressed topological feature vector is indexed and encoded by a local sensitive hashing algorithm to obtain a compressed index structure; wherein the local sensitive hashing algorithm adopts the SimHash algorithm, and the hash code length is 128 bits.

[0103] Specifically, in a specific embodiment, the process of performing graph embedding representation on the hierarchical index tree to obtain the index graph embedding vector is to convert the complex tree structure into a mathematical expression in numerical form. This conversion allows computers to process and analyze these structures more efficiently. For example, in a book classification system of a large library, the hierarchical index tree may represent a hierarchical relationship from the broadest category (such as natural sciences, humanities) to more specific subcategories (such as biology, physics, literary theory) and then to the location of a specific book. Through graph embedding representation, this process converts each node and the connection (edge) between them into a vector form, where the node vector dimension is set to 128 and the edge vector dimension is 64. This means that each node is described by a vector of 128 numbers, and each edge is characterized by a vector of 64 numbers. Once these index graph embedding vectors are obtained, the next step is to build a topological skeleton map based on them. In the example of this library, the topological skeleton map will contain those nodes that are crucial to understanding the entire classification system (i.e., skeleton nodes), as well as the paths connecting these key nodes (i.e., skeleton edges). In other words, it emphasizes which books or categories have direct relationships, while ignoring less important details. The result of this step is a simplified graph that retains the core information of the original structure, in which the key index nodes represent representative or frequently visited categories, and the key index paths are the routes that users often browse. In order to further refine the important features in the topological skeleton graph, the Persistent Homology algorithm is used to extract topological features. Persistent Homology is a mathematical tool used to study the shape and connectivity of data sets. It can reveal the topological properties of objects at different scales and present them in the form of persistent barcodes. A persistent barcode is actually a series of line segments, each of which represents the existence time of a specific topological feature - that is, the length of time that the feature remains unchanged throughout the scale change process. In the library scenario, this means that we can identify which classification patterns are stable at different levels of refinement, which helps us better understand and optimize the index structure. Then, based on the persistent barcode obtained above, we start to build a topological fingerprint vector and perform a nonlinear transformation on it to obtain a compressed topological feature vector. The topological fingerprint vector mentioned here is formed by encoding the persistent barcode. It also has 256 dimensions, but it is constructed by the MinHash algorithm. MinHash is a technique used to estimate the similarity of a set, which can help us maintain the original similarity while reducing the dimension. In our case, this means that even if we condense the information of the index tree into a shorter vector, we can still preserve the relative relationship between the various parts.Next, the purpose of nonlinear transformation is to make this feature vector better adapt to subsequent computing requirements, such as improving retrieval efficiency or enhancing noise resistance. Finally, in order to achieve fast and efficient query, it is necessary to index and encode the compressed topological feature vector through the local sensitive hashing algorithm to finally form a compressed index structure. In this process, SimHash is used as a variant of the local sensitive hashing algorithm, which can generate a hash code with a fixed length of 128 bits. The characteristic of SimHash is that it can ensure that similar objects produce similar hash values, which is particularly useful for approximate nearest neighbor search. Back to the application scenario of the library, this means that when a reader wants to find a book related to a certain topic, the system can quickly locate the closest candidates by comparing hash codes instead of traversing the entire database. This not only improves the search speed, but also reduces the storage space requirements, because the longer original feature vector has been converted into a short binary string. In summary, starting from the graph embedding representation of the hierarchical index tree to obtaining the compressed index structure through the local sensitive hashing algorithm, this series of steps constitutes a complete process aimed at ensuring the efficiency and accuracy of the indexing system. The work at each stage is to preserve the important information of the original structure as much as possible while minimizing the complexity and resource consumption. This method can not only be applied to static indexing systems such as libraries, but can also be extended to dynamically changing data environments, such as social network analysis, gene sequence comparison and other fields. As long as it involves complex hierarchical relationships and efficient retrieval needs, this solution can play its unique advantages.

[0104] In a specific embodiment, allocating and storing the original data in a storage area corresponding to the target memory based on the compressed index structure includes:

[0105] Obtain a distributed storage resource map of the current storage chip, perform dynamic resource scheduling on the original data based on the distributed storage resource map to obtain a storage resource scheduling strategy, and perform topology-aware mapping on the storage resource scheduling strategy to obtain a storage location mapping table,

[0106] Distributed data sharding is established based on the storage location mapping table to obtain a data sharding scheme, and an optimal transmission path is calculated for the original data based on the data sharding scheme to obtain a data transmission topology diagram;

[0107] Perform elastic scaling control on the data transmission topology diagram to obtain a dynamic allocation solution;

[0108] The original data is allocated and stored in a storage area corresponding to the target memory based on the dynamic allocation scheme.

[0109] Specifically, the process of allocating and storing the original data in the storage area corresponding to the target memory based on the compressed index structure is a key step in the edge computing storage method to ensure efficient data management and access. To achieve this, the system first needs to obtain the distributed storage resource map of the current storage chip, which is a comprehensive description of all available storage nodes and their performance characteristics in the entire network. This map not only shows the location, capacity, read and write speed and other information of each node, but also reflects the connection relationship and communication delay between them. For example, in an intelligent traffic monitoring system, each node in the urban area covered by the camera network may be equipped with different types of storage media, such as solid-state drives (SSDs), traditional hard disks (HDDs) or other forms of cache devices. By constructing such a distributed storage resource map, the system can clearly understand which nodes are suitable for storing hot data (frequently accessed data) and which are more suitable for cold data (less frequently accessed historical data). Based on the above distributed storage resource map, the system will dynamically schedule the original data to develop the best storage resource scheduling strategy. Dynamic resource scheduling means that the system can flexibly adjust the storage location of the data according to real-time needs to ensure that existing resources can be fully utilized at any time to avoid local overload or idle waste. Continuing with the example of intelligent transportation, when a large amount of video data is influxed at a certain intersection due to a sudden traffic accident, the system can quickly find nearby underutilized nodes through dynamic resource scheduling to temporarily expand the storage space; and when these emergencies are resolved, it can promptly restore to the conventional storage layout to maintain the overall balance of the system. Next, in order to make this resource scheduling more intelligent, the system will further perform topology-aware mapping on the storage resource scheduling strategy to obtain a storage location mapping table. Topology-aware mapping refers to optimizing the selection of data storage locations based on the consideration of the physical network topology structure to improve data access efficiency and reduce transmission costs. In this process, the system will comprehensively evaluate factors such as the distance between each node, bandwidth, and potential failure risks to find the most suitable "home" for each piece of data. For example, in a distributed camera network, the central server may give priority to nodes close to the data source and with high-bandwidth connections to save relevant data of key intersections or important events, while the edge nodes distributed everywhere are configured with an appropriate amount of local cache space for temporarily storing temporarily generated data. In this way, even in the case of complex and changing network conditions, each piece of data can be quickly located and accessed. Subsequently, the system establishes distributed data sharding based on the storage location mapping table to obtain a data sharding scheme. This operation is designed to break down large data sets into multiple smaller parts for parallel processing and decentralized storage. For intelligent traffic monitoring networks, this means that data from the same camera or cameras in adjacent areas can be divided into several small batches, each of which contains a snapshot of traffic conditions within a specific time period.Doing so can not only reduce the pressure on a single node, but also improve the response speed of the overall system. Then, the system will calculate the optimal transmission path for the original data based on the data sharding scheme to obtain a data transmission topology map. This step is to determine the best route from the data source to the target storage location to ensure that the delay and packet loss are minimized during the data transmission process. For example, when faced with a complex topology in an urban road network, the system can use algorithmic analysis to find out those paths with higher reliability and lower latency as the main channel for data transmission. As the data transmission topology map is generated, the system will perform elastic scaling control on the data transmission topology map to obtain a dynamic allocation plan. Elastic scaling control is the ability to automatically adjust resource configuration according to actual load conditions. It allows the system to add additional storage or computing resources during peak periods and release resources that are no longer used during trough periods, so as to always maintain efficient operation. In an intelligent transportation environment, this means that even if the amount of data on certain sections of the road suddenly increases, the system can quickly find a suitable storage location through an elastic scaling mechanism and start a backup node to provide support when necessary; in daily operations, data migration operations are performed regularly to gradually transfer cold data that is no longer frequently used to more economical storage media. This flexibility not only improves the adaptability of the system, but also reduces operation and maintenance costs. Finally, based on the dynamic allocation scheme, the system allocates and stores the original data in the storage area corresponding to the target memory. This final step ensures that after a series of carefully designed processing flows, each piece of data can be reasonably organized and efficiently stored in an appropriate physical location. For example, in an intelligent traffic monitoring system, video stream data after a series of operations such as feature extraction, hierarchical recursive segmentation, index tree construction and compression can be accurately mapped to the most suitable storage unit based on its multidimensional features and spatiotemporal attributes. This not only optimizes resource utilization, but also provides a solid foundation for subsequent data retrieval. In this way, even in the case of growing massive data, the system can still maintain stable performance and provide users with a more stable and reliable service experience. In summary, through a series of operations such as dynamic resource scheduling, topology-aware mapping, distributed data sharding establishment, optimal transmission path calculation, elastic scaling control, etc. on the distributed storage resource graph of the current storage chip, the system realizes effective management and optimized storage of original data. This process not only ensures that data can be placed in the right place at the right time, but also significantly improves query efficiency and service quality. For example, in an intelligent traffic monitoring system, managers can more quickly retrieve historical video records of a certain intersection within a specific time period, or track the driving trajectory of a vehicle throughout the city in real time. Not only that, this method also provides a useful technical reference for other similar large-scale data processing applications, promoting the development and innovation of edge computing technology.By continuously improving and perfecting data allocation and storage strategies, the system can better serve user needs in the increasingly complex IoT environment and provide a more intelligent and personalized service experience.

[0110] In a specific embodiment, the calculating the optimal transmission path of the original data based on the data sharding scheme to obtain a data transmission topology diagram includes:

[0111] Performing a topological complexity analysis on the data sharding scheme to obtain sharding transmission constraints, and performing a multi-objective optimization solution on the sharding transmission constraints to obtain transmission cost data, wherein the transmission cost data includes bandwidth consumption, delay overhead, and energy loss;

[0112] Performing distributed routing planning on the original data based on the transmission cost data to obtain a candidate path set, and performing annealing search on the candidate path set to obtain a local optimal path sequence;

[0113] Performing Pareto front analysis on the local optimal path sequence by a non-dominated sorting algorithm to obtain a multi-criteria evaluation index, and performing fuzzy hierarchical analysis on the multi-criteria evaluation index to obtain a path priority table;

[0114] Constructing a transmission topology based on the path priority table to obtain an initial transmission network, and performing hypergraph segmentation processing on the initial transmission network to obtain subnetwork clusters;

[0115] Performing collaborative game optimization on the sub-network cluster to obtain a Nash equilibrium solution set, and performing topological entropy calculation on the Nash equilibrium solution set to obtain an entropy value sequence;

[0116] The sub-network cluster is reconstructed for topological transmission based on the entropy value sequence to obtain a data transmission topology graph.

[0117] Specifically, the process of calculating the optimal transmission path of the original data based on the data sharding scheme and obtaining the data transmission topology diagram is an important part of the entire edge computing storage method. This process not only needs to consider the efficient transmission of data shards between different nodes, but also needs to ensure the stability and resource utilization of the overall network. To achieve this, the system first performs a topological complexity analysis on the data sharding scheme to determine the sharding transmission constraints. Topological complexity analysis aims to evaluate the various limitations and challenges that data shards may encounter when transmitting in the network, including but not limited to the physical distance between nodes, communication bandwidth, potential failure points, etc. For example, in an intelligent traffic monitoring system, when a camera captures an important traffic accident video, the video will be divided into multiple data shards, and each shard needs to find an optimal path from the current node to the target storage node. Through topological complexity analysis, the system can identify which paths may have high latency or low bandwidth problems, thereby providing a basis for subsequent optimization. Next, the system will perform multi-objective optimization based on the obtained sharding transmission constraints to obtain transmission cost data. Multi-objective optimization here refers to considering multiple performance indicators at the same time, such as bandwidth consumption, delay overhead and energy loss, in order to find a solution that can achieve the best balance in all aspects. In the intelligent transportation environment, this means that the system should not only minimize the network bandwidth occupied during data transmission, but also ensure that the transmission time is as short as possible and minimize the energy consumption of the device. By comprehensively weighing these factors, the system can generate a set of transmission cost data with detailed information, indicating the most economical and efficient transmission method for each data fragment. Based on the above transmission cost data, the system begins to perform distributed routing planning on the original data to obtain a set of candidate paths. Distributed routing planning is a method of dynamically selecting the optimal path in the network. It allows each node to make decisions independently while ensuring global consistency and coordination. At this stage, the system will comprehensively consider all possible transmission paths and their corresponding costs, and select a series of qualified candidate paths from them. Then, in order to further improve the search efficiency and find a better solution, the system will perform annealing search on the candidate path set to obtain a local optimal path sequence. The annealing algorithm is a heuristic algorithm that is particularly suitable for solving complex combinatorial optimization problems. It can explore a large number of possibilities in a short time and finally converge to a result close to the global optimal solution. For intelligent traffic monitoring networks, this step can help the system quickly locate paths that meet the transmission cost requirements and provide good service quality. As the local optimal path sequence is determined, the system will perform a Pareto front analysis on it through a non-dominated sorting algorithm to obtain a multi-criteria evaluation index. The non-dominated sorting algorithm is used to deal with multi-objective optimization problems. It can find a set of solutions that are not dominated by other options in multiple dimensions, the so-called Pareto front.In the intelligent transportation application scenario, this means that the system can select from a large number of local optimal paths those that are balanced in terms of bandwidth consumption, delay overhead, and energy loss. Then, the system will perform fuzzy hierarchical analysis on these multi-criteria evaluation indicators to obtain a path priority table. Fuzzy hierarchical analysis is a decision support tool that combines expert knowledge and mathematical models. It can assign corresponding weight values ​​according to the importance of different factors, thereby helping the system to more accurately judge the relative advantages of each path. For example, some sections may be considered more important due to their special geographical location. Even if their transmission costs are slightly higher than other paths, they still occupy a higher position in the overall evaluation. With the path priority table, the system can construct the transmission topology based on it to obtain the initial transmission network. This initial transmission network represents the best transmission path arrangement for all data fragments under current conditions, but it is not the final result and needs to be further optimized. To this end, the system will perform hypergraph segmentation on the initial transmission network to obtain subnetwork clusters. Hypergraph segmentation is a technique that decomposes a complex network into several smaller parts. It can simplify the problem scale and facilitate subsequent operations. In the context of intelligent transportation, this means that the transmission network within the entire city can be reasonably divided into multiple areas, each of which contains only a few key nodes and connection paths. This not only reduces the computational burden of the system, but also better adapts to the characteristics of the actual network structure. Finally, the system will perform collaborative game optimization on the sub-network clusters to obtain the Nash equilibrium solution set. Collaborative game optimization is a method of finding the optimal solution by simulating competition and cooperation behaviors. It is particularly suitable for solving problems involving conflicts of interest among multiple parties. In this process, each sub-network will negotiate with each other and try to find a strategy that can meet its own needs without affecting others. Specifically, in an intelligent transportation monitoring network, nodes in adjacent areas may discuss how to allocate limited bandwidth resources until they reach a consensus. Subsequently, the system will calculate the topological entropy of the Nash equilibrium solution set to obtain an entropy value sequence. Topological entropy is an indicator of the complexity of the network structure. It reflects the connection relationship between nodes within the network and its changing trend. By calculating the entropy value sequence, the system can evaluate the stability and reliability of each sub-network after optimization. Finally, based on the entropy value sequence, the system will perform topological transmission reconstruction on the sub-network cluster to obtain a data transmission topology map. This final data transmission topology diagram not only shows the transmission paths of all data shards, but also ensures that the entire network can maintain good performance in the face of various uncertainties. In summary, through a series of precise analysis, optimization and reconstruction of the data sharding scheme, the system achieves effective management and efficient transmission of the original data. This process not only greatly improves the speed and accuracy of data transmission, but also ensures the full utilization of network resources and the continuous improvement of service quality.For example, in an intelligent traffic monitoring system, managers can more quickly retrieve historical video records of a certain intersection within a specific time period, or track the driving trajectory of a vehicle throughout the city in real time. Not only that, this method also provides a useful technical reference for other similar large-scale data processing applications, and promotes the development and innovation of edge computing technology. By continuously improving and perfecting the transmission path calculation algorithm, the system can better serve user needs in an increasingly complex IoT environment and provide a more intelligent and personalized service experience.

[0118] The above describes the edge computing storage method based on the multi-layer stacking structure in the embodiment of the present invention. The following describes the edge computing storage system based on the multi-layer stacking structure in the embodiment of the present invention. Figure 2 In an embodiment of the present invention, an edge computing storage system based on a multi-layer stacking structure includes:

[0119] The extraction module 21 is used to perform distributed feature extraction on the original data of the preset storage chip to obtain multi-dimensional data features;

[0120] A segmentation module 22, configured to perform hierarchical and recursive segmentation on the original data based on the multi-dimensional data features to obtain multi-layer stacked data blocks;

[0121] A construction module 23 is used to construct a distributed index on the multi-layer stacked data blocks by using a preset heterogeneous fusion algorithm to obtain a hierarchical index tree;

[0122] A mapping module 24, configured to perform topological compression mapping on the hierarchical index tree to obtain a compressed index structure;

[0123] The allocation module 25 is used to allocate and store the original data in a storage area corresponding to the target memory based on the compressed index structure.

[0124] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the above method embodiment, which will not be repeated here.

[0125] Reference Figure 3 The present invention also provides a computer device in an embodiment, wherein the internal structure of the computer device can be as follows: Figure 3As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0126] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0127] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0129] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0130] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An edge computing storage method based on a multi-layer stacking structure, characterized in that: Applied to memory chips, including the following steps: Perform distributed feature extraction on the original data of the preset storage chip to obtain multi-dimensional data features; Based on the multi-dimensional data features, the original data is recursively divided into layers to obtain multi-layer stacked data blocks; Performing distributed index construction on the multi-layer stacked data blocks through a preset heterogeneous fusion algorithm to obtain a hierarchical index tree; Performing topological compression mapping on the hierarchical index tree to obtain a compressed index structure; Allocating and storing the original data in a storage area corresponding to the target memory based on the compressed index structure; The distributed index construction is performed on the multi-layer stacked data blocks by using a preset heterogeneous fusion algorithm to obtain a hierarchical index tree, including: Performing heterogeneous feature extraction on the multi-layer stacked data blocks to obtain multimodal feature vectors, and performing cross-modal similarity calculation on the multimodal feature vectors to obtain heterogeneous data association data; Performing local sensitive hash coding on the heterogeneous data association data to obtain a distributed hash index table; Using a distributed consistent hashing algorithm to perform load balancing mapping on the distributed hash index table to obtain a balanced index shard set, and constructing a Bloom filter on the balanced index shard set to obtain a fast query index; Constructing a multi-level index hierarchy based on the fast query index, and performing B+ tree conversion on the multi-level index hierarchy to obtain a hybrid index structure; Distributed segmentation of the hybrid index structure is performed using a graph partitioning algorithm to obtain a subgraph index set, and a minimum spanning tree is constructed for the subgraph index set to obtain an index tree skeleton; Performing index tree pruning and hierarchical index optimization on the index tree skeleton to obtain a hierarchical index tree; wherein the hierarchical index tree includes node identifiers, hierarchical relationships, data block locations, and index mapping relationships; The topological compression mapping of the hierarchical index tree to obtain a compressed index structure includes: Performing graph embedding representation on the hierarchical index tree to obtain an index graph embedding vector; wherein the index graph embedding vector includes a node vector and an edge vector, the node vector dimension is 128, and the edge vector dimension is 64; Constructing a topological skeleton graph based on the index graph embedding vector; wherein the topological skeleton graph comprises skeleton nodes and skeleton edges, the skeleton nodes represent key index nodes, and the skeleton edges represent key index paths; Extracting topological features from the topological skeleton map using a Persistent Homology algorithm to obtain a persistent barcode; wherein the persistent barcode represents the duration of topological features at different scales; Constructing a topological fingerprint vector based on the persistent barcode, and performing a nonlinear transformation on the topological fingerprint vector to obtain a compressed topological feature vector; wherein the topological fingerprint vector dimension is 256 and is constructed using a MinHash algorithm; The compressed topological feature vector is indexed and encoded by a local sensitive hashing algorithm to obtain a compressed index structure; wherein the local sensitive hashing algorithm adopts the SimHash algorithm, and the hash code length is 128 bits.

2. The edge computing storage method based on a multi-layer stacking structure according to claim 1 is characterized in that: The distributed feature extraction is performed on the original data of the preset storage chip to obtain multi-dimensional data features, including: Using the distributed computing framework of the storage chip, the original data of the preset storage chip is initially decomposed to obtain edge feature data; wherein the original data comes from the original data collected by different types of sensors; Slice the edge feature data to obtain a data slice set; Performing feature dimensionality reduction on the data shard set by using a distributed local sensitive hashing algorithm to obtain a reduced dimensionality data shard set; The cross-sensor feature fusion is performed on the dimension-reduced data slice set through a preset feature fusion algorithm to obtain multi-dimensional data features.

3. The edge computing storage method based on a multi-layer stacking structure according to claim 1 is characterized in that: The step of recursively dividing the original data into layers based on the multi-dimensional data features to obtain multi-layer stacked data blocks includes: Adaptively quantize and segment the multidimensional data features to obtain a feature segmentation sequence, and perform nonlinear boundary detection on the feature segmentation sequence to obtain a hierarchical boundary set; Based on the hierarchical boundary set, the original data is hierarchically and recursively segmented to obtain an initial data block set, and a tensor decomposition operation is performed on the initial data block set to obtain a multi-dimensional tensor block sequence; Performing spatiotemporal correlation analysis on the multidimensional tensor block sequence through a recursive neural network to obtain a correlation feature map between the original data, and performing hierarchical clustering processing on the correlation feature map to obtain a multi-level data block group; Reconstructing the topological structure of the multi-level data block group to obtain a hierarchical data block network, and dynamically balancing and optimizing the hierarchical data block network to obtain a balanced data block sequence; Performing inter-block relationship reasoning on the balanced data block sequence through a graph neural network to obtain a data block association graph, and performing edge weight calculation on the data block association graph to obtain a weighted data block set; The original data is reorganized into a stacked structure based on the weighted data block set to obtain multi-layer stacked data blocks.

4. The edge computing storage method based on a multi-layer stacking structure according to claim 1 is characterized in that: The allocating and storing the original data in a storage area corresponding to the target memory based on the compressed index structure comprises: Obtain a distributed storage resource map of the current storage chip, perform dynamic resource scheduling on the original data based on the distributed storage resource map to obtain a storage resource scheduling strategy, and perform topology-aware mapping on the storage resource scheduling strategy to obtain a storage location mapping table, Distributed data sharding is established based on the storage location mapping table to obtain a data sharding scheme, and an optimal transmission path is calculated for the original data based on the data sharding scheme to obtain a data transmission topology diagram; Perform elastic scaling control on the data transmission topology diagram to obtain a dynamic allocation solution; The original data is allocated and stored in a storage area corresponding to the target memory based on the dynamic allocation scheme.

5. The edge computing storage method based on a multi-layer stacking structure according to claim 4 is characterized in that: The calculating the optimal transmission path of the original data based on the data sharding scheme to obtain a data transmission topology diagram includes: Performing a topological complexity analysis on the data sharding scheme to obtain sharding transmission constraints, and performing a multi-objective optimization solution on the sharding transmission constraints to obtain transmission cost data, wherein the transmission cost data includes bandwidth consumption, delay overhead, and energy loss; Performing distributed routing planning on the original data based on the transmission cost data to obtain a candidate path set, and performing annealing search on the candidate path set to obtain a local optimal path sequence; Performing Pareto front analysis on the local optimal path sequence by a non-dominated sorting algorithm to obtain a multi-criteria evaluation index, and performing fuzzy hierarchical analysis on the multi-criteria evaluation index to obtain a path priority table; Constructing a transmission topology based on the path priority table to obtain an initial transmission network, and performing hypergraph segmentation processing on the initial transmission network to obtain subnetwork clusters; Performing collaborative game optimization on the sub-network cluster to obtain a Nash equilibrium solution set, and performing topological entropy calculation on the Nash equilibrium solution set to obtain an entropy value sequence; The sub-network cluster is reconstructed for topological transmission based on the entropy value sequence to obtain a data transmission topology graph.

6. An edge computing storage device based on a multi-layer stacking structure, characterized in that: Applied to memory chips, including: An extraction module is used to perform distributed feature extraction on the raw data of the preset storage chip to obtain multi-dimensional data features; A segmentation module, used for performing hierarchical and recursive segmentation on the original data based on the multi-dimensional data features to obtain multi-layer stacked data blocks; A construction module, used to perform distributed index construction on the multi-layer stacked data blocks through a preset heterogeneous fusion algorithm to obtain a hierarchical index tree; A mapping module, used for performing topological compression mapping on the hierarchical index tree to obtain a compressed index structure; An allocation module, configured to allocate and store the original data in a storage area corresponding to a target memory based on the compressed index structure; The distributed index construction is performed on the multi-layer stacked data blocks by using a preset heterogeneous fusion algorithm to obtain a hierarchical index tree, including: Performing heterogeneous feature extraction on the multi-layer stacked data blocks to obtain multimodal feature vectors, and performing cross-modal similarity calculation on the multimodal feature vectors to obtain heterogeneous data association data; Performing local sensitive hash coding on the heterogeneous data association data to obtain a distributed hash index table; Using a distributed consistent hashing algorithm to perform load balancing mapping on the distributed hash index table to obtain a balanced index shard set, and constructing a Bloom filter on the balanced index shard set to obtain a fast query index; Constructing a multi-level index hierarchy based on the fast query index, and performing B+ tree conversion on the multi-level index hierarchy to obtain a hybrid index structure; Distributed segmentation of the hybrid index structure is performed using a graph partitioning algorithm to obtain a subgraph index set, and a minimum spanning tree is constructed for the subgraph index set to obtain an index tree skeleton; Performing index tree pruning and hierarchical index optimization on the index tree skeleton to obtain a hierarchical index tree; wherein the hierarchical index tree includes node identifiers, hierarchical relationships, data block locations, and index mapping relationships; The topological compression mapping of the hierarchical index tree to obtain a compressed index structure includes: Performing graph embedding representation on the hierarchical index tree to obtain an index graph embedding vector; wherein the index graph embedding vector includes a node vector and an edge vector, the node vector dimension is 128, and the edge vector dimension is 64; Constructing a topological skeleton graph based on the index graph embedding vector; wherein the topological skeleton graph comprises skeleton nodes and skeleton edges, the skeleton nodes represent key index nodes, and the skeleton edges represent key index paths; Extracting topological features from the topological skeleton map using a Persistent Homology algorithm to obtain a persistent barcode; wherein the persistent barcode represents the duration of topological features at different scales; Constructing a topological fingerprint vector based on the persistent barcode, and performing a nonlinear transformation on the topological fingerprint vector to obtain a compressed topological feature vector; wherein the topological fingerprint vector dimension is 256 and is constructed using a MinHash algorithm; The compressed topological feature vector is indexed and encoded by a local sensitive hashing algorithm to obtain a compressed index structure; wherein the local sensitive hashing algorithm adopts the SimHash algorithm, and the hash code length is 128 bits.

7. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images