A data storage system, method, and storage medium based on edge computing

Abstract

This invention relates to the field of hard disk technology, and solves the technical problems of low global storage utilization in existing technologies, unnecessary redundancy waste when node reliability is high, and inability to provide sufficient fault tolerance when node reliability is insufficient. In particular, it relates to a data storage system, method, and storage medium based on edge computing. The method involves the following steps: collecting the raw data stream of the target data device, and obtaining preprocessed data blocks through data preprocessing based on the raw data stream. This invention uses consistent hashing to map nodes and data blocks to a ring space, providing a structured and scalable distributed basis for replica placement. It calculates the data reliability benefits under different replica numbers through an exponential decay availability model, and solves the Nash equilibrium point through a particle swarm optimization algorithm, minimizing global storage costs while ensuring availability, effectively solving the problem of excessive data redundancy between edge nodes.

Description

Technical Field

[0001] This invention relates to the field of hard disk technology, and in particular to a data storage system, method and storage medium based on edge computing. Background Technology

[0002] Hard disk related technologies refer to the hardware and software technologies in computer system architecture that perform data access, addressing and allocation, cache management and input / output control of storage devices such as hard disks. Data storage methods for edge computing based on hard disk related technologies involve collecting data at edge nodes and then directly uploading it to the central cloud for centralized storage. Data redundancy is achieved using a preset fixed number of replicas. Simple consistent hashing is used without considering the remaining space of the nodes. Hotness is determined at the memory layer using LRU or LFU algorithms, and a separate relational database is used to maintain the data index.

[0003] Existing technologies lack consideration for cost factors such as remaining node space and storage unit price. Redundancy strategies with a fixed number of replicas cannot perceive dynamic changes, and the system still maintains a two-replica configuration. Once the primary node and replica nodes go offline simultaneously due to link interruption, critical production data cannot be recovered. For example, in an edge computing scenario in an offshore oil field, some nodes are deployed on an unmanned platform and rely on satellite communication to synchronize with the shore. The stability of the communication link changes drastically with the weather. When the probability of communication interruption increases during the typhoon season, the system still maintains a two-replica configuration. Once the primary node and replica nodes go offline simultaneously due to link interruption, critical production data cannot be recovered. However, when the sea is calm, the communication link is highly reliable, and a two-replica configuration is sufficient. The system cannot dynamically reduce the number of replicas to free up storage space. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a data storage system, method, and storage medium based on edge computing. It solves the technical problems of existing technologies that lack consideration of cost factors such as node remaining space and storage unit price, resulting in low global storage utilization, unnecessary redundancy waste when node reliability is high, and inability to provide sufficient fault tolerance when node reliability is insufficient.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a data storage method based on edge computing, the steps of which are: collecting the raw data stream of the target data device, and obtaining preprocessed data blocks by data preprocessing based on the raw data stream; Based on the preprocessed data blocks, the heat load is predicted to generate write pressure prediction values. Threshold clustering is performed on the write pressure prediction values ​​to find the target storage area. Based on the target storage area, the health index is used to build and process the data to generate an optimized write data index. Behavioral sequence features are embedded into the optimized write data index to generate data block embedding vectors. Cooperative sequence processing is performed based on the data block embedding vectors to generate temporal feature vectors. The temporal feature vectors are then processed through an attention weight candidate set to generate prefetch cache data. Based on the prefetched cache data, the master node identifier is obtained through node affiliation processing. Virtual regression cost processing is performed based on the master node identifier to obtain the cost function. Availability balancing processing is then performed on the cost function to generate the globally optimal data storage state. The final hard disk data storage state is obtained by dynamically partitioning and storing data based on the globally optimized data storage state.

[0006] Preferably, data preprocessing is performed based on the original data stream, including: extracting structured data frames by parsing the original data stream through protocol parsing; Sliding outlier processing is performed based on structured data frames to generate effective data payloads; The effective data payload is divided into preprocessed data blocks by a fixed duration.

[0007] Preferably, based on the preprocessed data blocks, the heat load is predicted to generate write pressure prediction values, and the write pressure prediction values ​​are used for threshold clustering location lookup, including: processing the preprocessed data blocks through data block heat labeling to obtain heat label data blocks; Short-term load prediction is performed based on heat tag data blocks to generate write pressure prediction values; Threshold clustering is performed on the predicted pressure values ​​to generate pressure level codes. The target storage area is obtained by performing a hash location lookup based on the pressure level code.

[0008] Preferably, the process of building a health index based on the target storage region includes: Based on the target storage region, write throttling is performed using health status to obtain optimized write commands; The optimized write commands are indexed to generate an optimized write data index.

[0009] Preferably, behavioral sequence features are embedded into the optimized write data index to generate a data block embedding vector, and collaborative sequence processing is performed based on the data block embedding vector, including: constructing a user access vector based on the optimized write data index through user behavior sequences; Data block feature embedding is performed based on user access vectors to generate data block embedding vectors; Perform co-similarity processing on the data block embedding vectors to obtain a list of similar data blocks; A time-series feature vector is generated by accessing the sequence based on a list of similar data blocks.

[0010] Preferably, the temporal feature vector is processed through an attention weight candidate set, including: Attention weight vectors are obtained by performing attention weight allocation based on temporal feature vectors. The attention weight vector is prefetched to generate prefetch cache data.

[0011] Preferably, the master node identifier is obtained by processing the node affiliation based on the prefetched cache data, and virtual regression cost processing is performed based on the master node identifier, including: obtaining the node hash ring by processing the node mapping based on the prefetched cache data; Data attribution is processed based on node hash rings to obtain the master node identifier; The primary node identifier is virtually mapped to generate a list of candidate replica nodes; The cost function is obtained by performing linear regression cost processing based on the candidate replica node list.

[0012] Preferably, the cost function is subjected to availability balancing processing, including: Data availability benefits are processed based on cost functions to generate benefit functions; The payoff function is subjected to particle swarm equilibrium processing to generate the globally optimal data storage state.

[0013] This technical solution also provides a data storage system based on edge computing, which includes: The preprocessing module is used to acquire the raw data stream from the target data device and obtain preprocessed data blocks based on the raw data stream through data preprocessing. The hash data index module is used to predict the heat load based on preprocessed data blocks, generate write pressure prediction values, perform threshold clustering location lookup on the write pressure prediction values ​​to obtain the target storage area, and generate an optimized write data index based on the target storage area through health index construction. The prefetch cache module is used to embed behavioral sequence features into the optimized write data index, generate data block embedding vectors, perform co-sequence processing based on the data block embedding vectors to generate temporal feature vectors, and process the temporal feature vectors through the attention weight candidate set to generate prefetch cache data. The global optimal storage module is used to obtain the master node identifier by processing the node affiliation of the prefetched cache data, perform virtual regression cost processing based on the master node identifier to obtain the cost function, perform availability balancing processing on the cost function, and generate the global optimal data storage status. The dynamic storage module is used to dynamically partition and store data based on the globally preferred data storage state to obtain the final hard disk data storage state.

[0014] According to one aspect of some embodiments of this disclosure, a computer-readable storage medium is provided having computer program instructions stored thereon that, when executed, implement the steps of any of the edge computing-based data storage methods mentioned above.

[0015] By employing the above technical solutions, the present invention provides a data storage system, method, and storage medium based on edge computing, which has at least the following beneficial effects: 1. This invention adopts a dynamic protocol identification and field extraction mechanism, which is compatible with the reporting formats of different edge devices, unifies heterogeneous data into structured data frames, introduces an outlier removal method based on sliding windows, effectively removes measurement noise and abnormal interference while retaining the real data change trend, and adopts a fixed duration window segmentation strategy, which facilitates subsequent hot and cold data identification and prediction model processing, avoids the overhead caused by repeated format conversion in traditional decentralized processing, and significantly improves the quality and efficiency of edge-side data preprocessing.

[0016] 2. This invention employs a differential autoregressive moving average model based on access frequency, which not only accurately quantifies the hotness or coldness of data blocks but also predicts short-term write load pressure in advance. Based on hash-based location lookup, it avoids load concentration, dynamically adjusts the write latency coefficient, and actively controls the write rate to balance partition wear. This allows the write position to adapt to dynamic changes in load, effectively reducing the head seek distance, balancing partition lifespan, suppressing write amplification at its source, and significantly improving the write efficiency and reliability of the hard drive in edge computing environments.

[0017] 3. This invention embeds the sequential information of data blocks in the access sequence into a feature representation in the form of trigonometric functions, enabling the model to focus on the preceding data blocks that are most decisive for subsequent accesses. It integrates the collaborative behavior features of user groups with the temporal dependency features of individual access sequences, improving processing efficiency and significantly enhancing the matching accuracy between prefetched content and actual access needs. By constructing user access vectors through deep collaborative filtering, it generates differentiated data block embeddings for different operators, which can effectively distinguish individual behavior patterns and avoid multi-user aliasing interference.

[0018] 4. This invention uses consistent hashing to map nodes and data blocks to a ring space, generating a uniformly distributed list of candidate replica nodes. This provides a structured and scalable distributed basis for replica placement. The replica number decision is modeled as a game theory problem. The data reliability benefits under different replica numbers are calculated through an exponential decay availability model. The Nash equilibrium point is solved through a particle swarm optimization algorithm. This minimizes the global storage cost while ensuring availability, effectively solving the problem of excessive data redundancy between edge nodes. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a data storage method based on edge computing according to the present invention; Figure 2 This is a structural block diagram of a data storage system based on edge computing according to the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0021] Current technologies lack consideration for cost factors such as remaining node space and storage unit price, resulting in low global storage utilization. This leads to unnecessary redundancy and waste when node reliability is high, while failing to provide sufficient fault tolerance when node reliability is insufficient. Please refer to [reference needed]. Figure 1 This embodiment provides a data storage method based on edge computing, which can improve the utilization of global storage, reduce unnecessary redundancy and waste, and dynamically adjust the use of storage space. The method includes the following steps: S1. Collect the raw data stream of the target data device, and obtain the preprocessed data block through data preprocessing based on the raw data stream; Existing technologies are often designed for a single data source and lack the unified access capability for multi-source heterogeneous data, which makes it difficult to fuse and process data from different types of sensors, and it is difficult to adapt to the dynamic changes in data distribution in the edge environment, which can easily lead to the problem of false rejection or missed rejection. To solve the above problems, the specific implementation steps are as follows: S11. Extract structured data frames from the raw data stream through protocol parsing. In this step, the system continuously captures the raw data stream, which consists of raw messages in binary or text format reported by different edge data devices. During processing, the system first performs dynamic matching and protocol identification on the data stream according to a preset feature library of multiple communication protocols, dividing the continuous bit stream into data frames with complete semantics. Then, the system extracts fields from the header and payload of each data frame. By calculating the check field at a specific offset position in the data frame, it is compared with the calculation result obtained by the polynomial cyclic redundancy check algorithm based on other data in the frame. Only when the two are completely consistent is the frame considered valid. Next, the system normalizes and transforms the discrete fields such as timestamps, device identifiers, and sensor values ​​carried in the frame. By mapping the values ​​of different fields to a unified internal representation format according to a preset data dictionary, and combining the scaling of the original measurement values ​​with linear transformation of the baseline offset, the system finally combines these processed fields into a structured data frame.

[0022] S12. Perform sliding outlier processing based on structured data frames to generate effective data payloads. In this step, the system first constructs a fixed-length sliding window for continuously arriving data points. The window moves forward step by step as new data flows in, ensuring that the dataset processed each time is a local sampling sequence within the most recent period. At each window position, the system first calculates the sum of all data points in the window, divides the sum by the number of data points in the window to obtain the arithmetic mean of the current window, and then the system calculates the difference between each data point and the mean. The system squares each difference and sums the results. Then, it divides this sum by the number of data points in the window. Finally, it takes the square root of the quotient to obtain the standard deviation of the data in the current window. Based on the three-standard-deviation criterion in statistics, the system evaluates each data point in the window one by one: it calculates the absolute difference between the data point and the mean. If the absolute difference is less than three standard deviations, the data point is considered valid and retained. Otherwise, it is discarded as an outlier. After the window slides through, the system sums all the retained valid data points and divides this sum by the number of valid data points to obtain the filtered valid data payload.

[0023] S13. The effective data payload is divided into fixed-duration segments to obtain preprocessed data blocks. In this step, the system first determines the boundary division point of each data block according to the preset time window length. The arrival time of the first effective data payload is taken as the start time. The start time is added to the time window length to obtain the end time boundary of the first window. Then, the system traverses all effective data payloads, extracts continuous data points with timestamps between the start time and the end time, and splices these data points in chronological order to form the first original data sequence. After the data extraction of the first window is completed, the system takes the end time of the previous window as the new start time, adds it to the time window length again to obtain the new end time, and so on, dividing the continuous time axis into several time segments of equal length. For the effective data payload in each time segment, the system calculates the time offset of each data point relative to the start time of the window, uses these offsets as relative time labels in the data frame, and encapsulates them together with the corresponding numerical fields into structured data units. The system concatenates all data units belonging to the same time segment in chronological order to finally generate a preprocessed data block of fixed duration.

[0024] This invention employs a dynamic protocol identification and field extraction mechanism, which is compatible with the reporting formats of different edge devices. It unifies heterogeneous data into structured data frames and introduces an outlier removal method based on a sliding window. This effectively removes measurement noise and abnormal interference while preserving the true data change trend. The fixed-duration window segmentation strategy facilitates subsequent hot and cold data identification and prediction model processing, avoiding the overhead caused by repeated format conversion in traditional decentralized processing, and significantly improving the quality and efficiency of edge-side data preprocessing.

[0025] S2. Based on preprocessed data blocks, perform hot load prediction to generate write pressure prediction values. Perform threshold clustering location lookup on the write pressure prediction values ​​to obtain the target storage area. Based on the target storage area, construct and process through health index to generate an optimized write data index. Existing technologies lack the ability to dynamically predict write load, causing writes to continue in the conventional way during high load periods, exacerbating the write amplification effect. They often use sequential writes or simple area isolation, which can easily lead to some areas being excessively worn while other areas are idle. When the access mode changes abruptly, it cannot be adjusted in time, causing the read / write head to frequently seek across areas, resulting in a decrease in overall write efficiency. To solve the above problems, the specific implementation steps are as follows: S21. Based on the preprocessed data blocks, the data blocks are processed by data block heat marking to obtain heat tag data blocks. In this step, the system first obtains the number of times each preprocessed data block is accessed within a unit time. This number is obtained by statistically analyzing the frequency of the data block appearing in the most recent unit time window in the historical access records. At the same time, the system records the time interval between the current time and the last time the data block was accessed. The difference between the two times is calculated to obtain the interval length in units of time. Then, the reciprocal of the interval length is taken to convert it into a value that is positively correlated with the access timeliness. The system calculates the above two values ​​with the preset weight coefficient: the access frequency value is multiplied by the weight coefficient to obtain the frequency contribution value. The difference between the first value and the weighting coefficient is used as the weight of the timeliness factor. Multiplying this by the timeliness value yields the timeliness contribution value. The system then adds the frequency contribution value and the timeliness contribution value to obtain the popularity value of the data block. This popularity value is a quantitative indicator that combines the frequency of access and the freshness of access time, reflecting the popularity of the data block at the current moment. Based on the preset popularity threshold range, the system maps the popularity value to the corresponding popularity level label and attaches the label to the original preprocessed data block, finally outputting a data block with a popularity label.

[0026] S22. Based on the heat label data block, perform short-term load prediction and generate write pressure prediction values. In this step, the historical write load sequence is first used as the analysis object. The stationarity of the sequence is tested. If the sequence is non-stationary, it is made stationary through difference operation. Then, the autoregressive part is constructed: select historical observations of a certain order, multiply each historical observation by its corresponding autoregressive coefficient, and then sum all the products to obtain the value of the autoregressive component. At the same time, the system constructs the moving average part: select historical error terms of a certain order, multiply each historical error term by its corresponding moving average coefficient, and then sum all the products to obtain the value of the moving average component. The constant term is added to the autoregressive and moving average components, and the white noise error term at the current moment is added to obtain the predicted write pressure value at the current moment. During the model fitting process, the system determines the values ​​of the autoregressive and moving average coefficients by maximizing the likelihood function or minimizing the sum of squared residuals, so that the model's fitting error to the historical sequence is minimized. The historical write load sequence consists of the amount of data actually written to the hard drive at each past moment, which is the time series data continuously accumulated during the system's operation. If the sequence has been determined to be stationary, the original sequence is directly used for parameter estimation and fitting of the autoregressive moving average model.

[0027] S23. Threshold clustering is performed on the predicted write pressure values ​​to generate a pressure level code. In this step, a large number of predicted write pressure values ​​accumulated during historical operation are first clustered. Several cluster centers are determined through iterative optimization. Each cluster center represents a typical write pressure level and corresponds to a unique pressure level code. During the clustering process, the squared Euclidean distances between all sample points and each cluster center are summed. The position of the cluster centers is repeatedly updated until convergence, with the goal of minimizing the sum. Finally, a set of cluster centers covering different pressure ranges from low to high is formed. When the predicted write pressure value at the current moment is received, the system calculates the difference between the predicted value and the value of each cluster center in the set in turn. Each difference is squared to obtain the squared distance. Then, the size of all squared distances is compared, and the cluster center corresponding to the minimum value is found. The pressure level code of the cluster center is used as the output to complete the mapping from continuous values ​​to discrete levels.

[0028] S24. Perform hash location lookup based on stress level encoding to obtain the target storage area. In this step, the input stress level encoding is first used as the input parameter of the hash function. The encoding is converted into a uniformly distributed hash value through hash calculation. The hash value exhibits randomization characteristics in the numerical space, which can effectively avoid adjacent stress levels from mapping to adjacent storage areas. Then, the total number of partitions pre-divided on the hard disk is obtained. The hash value is divided by the total number of partitions, and the remainder is used as the index value of the partition list. The index value is obtained by modulo operation between the hash value and the total number of partitions, ensuring that the output range always falls within the valid partition index range. Based on the calculated index value, the partition identifier at the corresponding position is retrieved from the hard disk partition list and identified as the target storage area.

[0029] S25. Based on the health status of the target storage area, write throttling is performed to obtain optimized write instructions. In this step, the total amount of data currently written to the target storage area and the maximum data capacity preset for the area are first obtained. The total amount written is divided by the maximum capacity to obtain a ratio reflecting the space occupancy of the area. At the same time, the system obtains the current operating temperature value of the area and the maximum operating temperature value that the area can reach. The current temperature is divided by the maximum allowable temperature to obtain a ratio reflecting the heat load status of the area. The preset balance factor is used as a weighting parameter, and the space occupancy ratio is multiplied by the balance factor to obtain the space contribution value. The difference between the heat load ratio and the balance factor is multiplied by 1 to obtain the heat load contribution value. Then, the space contribution value and the heat load contribution value are added to obtain the write latency coefficient of the target storage area. The write latency coefficient is a value between 0 and 1, which represents the current overall health status of the area. The larger the coefficient value, the heavier the area load and the lower the health. Optimized write instructions are generated based on the latency coefficient. The instructions include the latency waiting time of the write operation. The waiting time is positively correlated with the latency coefficient. That is, the lower the health of the area, the longer the write operation is delayed.

[0030] S26. The optimized write command is indexed to generate an optimized write data index. In this step, the unique identifier of the data block is first parsed from the optimized write command and used as the primary key of the index. Simultaneously, the physical address pointer of the data block actually written to the hard disk is obtained. The system uses these key-value pairs as the basic units of the index and inserts them into the tree index structure in numerical order according to the primary key. During insertion, the system starts from the root node and, based on the comparison result between the primary key and the key value within the node, locates the corresponding leaf node layer by layer. When a leaf node is not full, the system directly inserts the new key-value pair into the node while maintaining the ordered arrangement of the key values ​​within the node. When a leaf node is full, the system splits the node into two nodes, promotes the middle key value to the parent node, and recursively processes the splitting operation of the parent node until the tree structure constraints are met. The system also maintains a doubly linked list between all leaf nodes, allowing range queries to be quickly traversed through the leaf node pointers. The entire tree structure remains balanced throughout the insertion process, ensuring that the path length from all leaf nodes to the root node is the same.

[0031] This invention employs a differential autoregressive moving average model based on access frequency, which not only accurately quantifies the hotness or coldness of data blocks, but also predicts short-term write load pressure in advance. For example, when an edge node receives a large amount of hot data in a short period of time, such as when an abnormal event in video surveillance triggers multiple simultaneous uploads, the system still writes all data to the same high-performance area according to a fixed strategy, causing the write load of that area to exceed the limit instantly, triggering write amplification. The read / write head repeatedly seeks the path due to frequent garbage collection, resulting in a sharp increase in write latency. Furthermore, the discretized level code is obtained through cluster analysis, providing a clear basis for subsequent allocation. Hash-based location lookup avoids load concentration, dynamically adjusts write latency coefficients, and actively controls write rates to balance partition wear. For example, if a partition is nearing its nominal write lifespan due to long-term hot data storage and poor heat dissipation causing persistently high temperatures, newly written hot data may still be allocated to this partition, accelerating its physical wear and causing bad blocks to appear prematurely before the partition's overall lifespan is exhausted, resulting in data loss. At the same time, it can also make the write position adaptively adjusted according to the load, effectively reducing the head seek distance, balancing partition lifespan, suppressing write amplification at the source, and significantly improving the write efficiency and reliability of the hard drive in edge computing environments.

[0032] S3. Optimize the written data index by embedding behavioral sequence features to generate data block embedding vectors. Based on the data block embedding vectors, perform collaborative sequence processing to generate temporal feature vectors. Based on the temporal feature vectors, process them through an attention weight candidate set to generate prefetch cache data. Existing collaborative filtering technologies cannot calculate the similarity between data blocks and other data blocks, resulting in the data block never being prefetched. Temporal prediction methods have weak modeling capabilities for the correlation between distant preceding and following items in the sequence. Methods based on fixed windows or frequency statistics cannot distinguish between short-term hotspots and long-term popular content, resulting in severe prefetch lag. For example, a camera on a certain road segment usually uploads ordinary traffic flow data periodically, but when a traffic accident occurs, it suddenly uploads a large amount of key frame data continuously. The prefetching strategy based on historical frequency still prefetches in the conventional mode and cannot switch to the accident-related data block sequence in time, causing a surge in key frame data access delay and affecting the response timeliness of the real-time analysis system. To solve the above problems, the specific steps are as follows: S31. Based on the optimized write data index, user access vectors are generated by constructing user behavior sequences. In this step, the data index after the disk optimization write is traversed first to extract all access relationship records between user identifiers and data block identifiers, and to determine the total number of users and data blocks in the system. For each user, a vector with a length equal to the total number of data blocks is created, and each dimension of the vector corresponds to a unique data block. Then, the access records of each data block by the user are checked one by one. If the user has accessed a data block, the value of the corresponding dimension of the data block is set to 1. If the user has never accessed the data block, the value of the corresponding dimension is set to 0. By performing the assignment operation on all dimensions in sequence, a binary vector consisting of 0 and 1 is generated for each user. The value of each dimension in this vector is determined only by whether the user has accessed the corresponding data block. This user access vector is used as the output, which accurately describes the historical interaction pattern between each user and the data block set in a high-dimensional sparse form, providing a structured input representation for subsequent data block feature embedding and collaborative similarity calculation.

[0033] S32. Based on the user access vector, embed the data block features to generate the data block embedding vector. In this step, we first receive the high-dimensional sparse access vector of each user as input. The dimension of the vector is equal to the total number of data blocks in the system. The value in each dimension indicates whether the user has accessed the corresponding data block. The system pre-trains a weight matrix and a bias vector. The number of rows in the weight matrix is ​​equal to the preset embedding dimension, the number of columns is equal to the total number of data blocks, and the length of the bias vector is equal to the embedding dimension. For the current input user access vector, perform matrix multiplication operation between the weight matrix and the user access vector. Specifically, each row of the weight matrix is ​​multiplied by the user access vector, i.e., the corresponding values ​​are multiplied and summed to obtain an intermediate vector. The length of this intermediate vector is equal to the embedding dimension. Then, the bias vector is added to the intermediate vector at corresponding positions, i.e., the value of each dimension of the bias vector is added to the value of the intermediate vector in the same dimension, to obtain the data block embedding vector. This embedding vector, as the output, is a low-dimensional dense continuous value vector. The value of each dimension is extracted from the original user access behavior through the above multiplication and addition operations, compressing the interaction pattern between the user and the data block into a compact vector space. This provides the feature representation of the data block in the semantic space for subsequent collaborative similarity calculation.

[0034] S33. Perform co-similarity processing on the data block embedding vectors to obtain a list of similar data blocks. In this step, first obtain the embedding vectors of the two data blocks. These two vectors have the same number of dimensions, and each dimension contains continuous values. Calculate the dot product of the two vectors, that is, multiply the values ​​of the corresponding dimensions of the two vectors respectively, and then sum all the product results to obtain a scalar value. At the same time, calculate the magnitude of each vector. For each vector, square the value of each dimension, sum all the square values, and then take the square root of the sum to obtain the magnitude of the vector. The dot product of the two vectors is then divided by the product of their magnitudes. The quotient is the cosine similarity between the two data blocks. This cosine similarity reflects the directional consistency of the two data blocks in the embedding space. The larger the value, the higher the correlation between the two data blocks in terms of user access patterns. Taking the current data block as the benchmark, the cosine similarity between it and all other data blocks is calculated in turn. All other data blocks are sorted in descending order of similarity value. Several data blocks with the highest ranking are selected, and their identifiers and corresponding similarity values ​​are organized together to form a list of similar data blocks.

[0035] S34. Based on the list of similar data blocks, a time-series feature vector is generated through access sequence processing. In this step, the position index of each data block in the sequence is first obtained from the list of similar data blocks. This index increases sequentially from the starting position. A fixed model dimension is preset as the length of the output vector. This dimension is even. For each position index in the sequence, a time-series feature vector with a length equal to the model dimension is generated. The value of each dimension in this vector is determined as follows: For positions with even dimension indices, the position index is divided by the dimension index of 10,000 and then by the power of the model dimension to obtain the intermediate value. The sine function of this intermediate value is then taken to obtain the value of this dimension. For positions with odd dimension indices, the power value is obtained by subtracting one from the dimension index and dividing by the model dimension. The position index is then divided by the power of 10,000 to obtain the intermediate value. The cosine function of this intermediate value is then taken to obtain the value of this dimension. The sine or cosine calculations of all dimensions are performed sequentially in ascending order of dimension index. The calculation results are arranged in order to finally form a complete time-series feature vector.

[0036] S35. Based on the temporal feature vector, attention weights are assigned to obtain attention weight vectors. In this step, the input temporal feature vector sequence is first processed by three different linear transformation matrices to generate corresponding query matrix, key matrix and value matrix. These three matrices have the same sequence length. The query matrix and key matrix have the same dimension for each vector, and the value matrix has the preset model dimension for each vector. The transpose of the query matrix and key matrix is ​​multiplied by matrix multiplication, that is, the dot product of each position vector in the query matrix and each position vector in the key matrix is ​​performed. That is, the corresponding dimension values ​​are multiplied and summed to obtain a square matrix. Each element in the square matrix represents the correlation strength between two positions in the sequence. Next, the dimension value of the key vector is obtained, and the square root of the value is taken. The square root is used as a scaling factor, and each element in the aforementioned square matrix is ​​divided by the scaling factor to complete the numerical scaling operation. The normalized exponential function is applied to the scaled square matrix row by row, that is, each value in each row is raised to the power of the natural constant, and all exponential values ​​in each row are summed. Each exponential value is divided by the sum of the row to obtain the normalized weight matrix. Finally, the weight matrix and the value matrix are multiplied by matrix multiplication, that is, each row of the weight matrix is ​​multiplied by the corresponding column of the value matrix to generate the final attention weight vector sequence.

[0037] S36. Perform prefetch candidate set processing on the attention weight vector to generate prefetch cache data. In this step, the attention weight vector is first received as input. Each element in the vector corresponds to the weight value of a data block. The larger the weight value, the stronger the correlation between the data block and the current access mode. The weight values ​​of all data blocks are sorted and the ranking position of each data block is determined according to the order of the values ​​from high to low. Then, the filtering threshold is determined according to the preset candidate set capacity parameter. This threshold specifies the number of data blocks to be selected. Starting from the data block with the highest weight value, if the attention weight is greater than or equal to the filtering threshold, it is included in the prefetch candidate set. Otherwise, it is not included. Then, the corresponding data block identifiers are added to the candidate set in turn until the number of selected data blocks reaches the preset capacity limit. For cases where the weight values ​​are the same, selection is performed according to the inherent order of the data block identifiers to ensure the determinism of the filtering process. During the entire filtering process, only data blocks that pass the threshold filtering are retained, and data blocks that are not selected are not included in the prefetch candidate set. This prefetch candidate set is the prefetch cache data. The prefetch cache data, as the output, is a set consisting of several data block identifiers. Each data block in the set has a high attention weight, representing the set of data blocks that are most likely to be accessed next in the current access mode. This provides prefetch data that needs to be loaded into the cache in advance for subsequent multi-edge node collaborative data redundancy optimization.

[0038] This invention introduces positional encoding into a list of similar data blocks, embedding the order information of data blocks in the access sequence into a feature representation in the form of trigonometric functions. Then, it calculates the association weights between different positions in the sequence through a scaled dot product attention mechanism, enabling the model to focus on the preceding data blocks that are most decisive for subsequent accesses. This allows the prefetching decision to simultaneously integrate the collaborative behavior characteristics of the user group and the temporal dependency characteristics of the individual access sequence. For example, if a diagnostic model file for a certain industrial equipment is newly deployed on an edge node, there are no access records initially. However, operators need to frequently retrieve the file for equipment debugging. Because the prefetching mechanism fails to recognize its importance, each access requires reading from the hard drive, causing a delay of seconds and affecting debugging efficiency. It significantly improves the matching accuracy between prefetched content and actual access needs, and effectively solves the problems of prefetching lag and cache pollution in traditional methods. For example, in edge video analysis scenarios, cameras on a certain road section usually periodically upload ordinary traffic flow data, but when a traffic accident occurs, a large number of key frame data are suddenly uploaded continuously. The prefetching strategy based on historical frequency still prefetches in the conventional mode and cannot switch to the data block sequence related to the accident in time, causing a surge in access delay for key frame data and affecting the response time of the real-time analysis system. Existing technologies train multi-user access sequences in a mixed manner, causing the prefetch model to be unable to distinguish the behavior patterns of different operators. For example, after operator A completes mold replacement, he needs to access the temperature calibration table, while operator B completes mold replacement and needs to access the pressure test script. Existing technologies mix and statistically analyze the subsequent accesses of the two operators, so that the prefetched content after mold replacement contains two possibilities at the same time. The cache space is scattered and occupied, and the actual hit rate decreases instead of increasing. This invention constructs user access vectors first through deep collaborative filtering and generates differentiated data block embeddings for different operators, which can effectively distinguish individual behavior patterns and avoid interference from multi-user aliasing.

[0039] S4. Based on the prefetched cache data, the master node identifier is obtained through node affiliation processing. Virtual regression cost processing is performed based on the master node identifier to obtain the cost function. Availability balancing processing is then performed on the cost function to generate the globally preferred data storage state. Existing technologies use a fixed number of replicas or simple consistent hash successor nodes, which lack consideration for cost factors such as the remaining space of nodes and the unit price of storage. This leads to popular nodes being filled up too early while other nodes are idle, resulting in low global storage utilization. Usually, a preset fixed number is used, which cannot be dynamically adjusted according to the node availability probability. This causes unnecessary redundancy waste when the node reliability is high, and cannot provide sufficient fault tolerance guarantee when the node reliability is insufficient. The existing distributed hash routing and replica number decision-making are independent, resulting in high-cost nodes and low-cost nodes being treated equally, making it impossible to achieve economically optimal differentiated placement. For example, in an edge computing scenario in an offshore oil field, some nodes are deployed on unmanned platforms and rely on satellite communication to synchronize with the shore. The stability of the communication link changes drastically with the weather, and the existing redundancy strategy with a fixed number of replicas cannot perceive this dynamic change: when the probability of communication interruption increases during the typhoon season, the system still maintains a two-replica configuration. Once the master node and replica nodes go offline due to link interruption, critical production data cannot be recovered. However, when the wind and waves are calm, the communication link is highly reliable, and a two-replica configuration is sufficient. The system cannot dynamically reduce the number of replicas to free up storage space. To solve the above problems, the specific steps are as follows: S41. Based on the prefetched cache data, a node hash ring is obtained through node mapping processing. In this step, the network protocol address and service port number of each node in the edge cluster are first obtained. These two pieces of information are concatenated into a string in a fixed order as the unique identifier of the node. Then, the string is input into a secure hash algorithm. After multiple rounds of iterative compression operations, a fixed-length hash value is generated. The hash value exhibits a uniform distribution characteristic in the numerical space. According to the preset hash ring space dimension, the total capacity of the hash ring is determined to be two raised to the power of that number. The hash value generated by each node is divided by the total capacity of the hash ring, and the remainder is taken as the mapping position of the node on the hash ring. This remainder calculation process ensures that the mapping positions of all nodes fall within the closed interval from zero to the total capacity of the hash ring minus one. The above concatenation, hash calculation and modulo operation are performed on all nodes in sequence to obtain the position identifier of each node on the hash ring. These position identifiers are organized into an ordered ring structure according to their numerical values, i.e., the node hash ring.

[0040] S42. Data attribution processing based on node hash ring to obtain master node identifier; In this step, the unique identifier of the data block to be stored is first obtained. This identifier is used as input and hashed using a secure hash algorithm to generate a fixed-length hash value. The total capacity of the hash ring is determined according to the spatial dimension of the hash ring. The hash value of the data block identifier is divided by the total capacity of the hash ring, and the remainder is taken as the mapping position of the data block on the hash ring. This remainder calculation ensures that the position of the data block falls within the range of zero to the total capacity of the hash ring minus one. Then, the mapping position of the data block is compared with the position identifiers of all nodes on the node hash ring. Starting from the mapping position, the search proceeds sequentially along the increasing direction of the hash ring value to find the first node position identifier that is greater than or equal to the data block mapping position. If no node that meets the condition is found in the increasing direction, the search continues from the minimum position of the hash ring, forming a circular search. The node corresponding to the finally found node position identifier is determined as the master node, and the unique identifier of the node is extracted as the master node identifier.

[0041] S43. Perform virtual mapping processing on the master node identifier to generate a candidate replica node list. In this step, first obtain the position identifier of the master node on the hash ring and the preset maximum number of replicas. At the same time, determine a fixed step size value. This step size is used to control the interval distance between replica nodes. Starting from the count value, it is incremented sequentially to the maximum number of replicas. For each count value, add the product of the master node identifier and the count value multiplied by the step size to obtain an offset position value. Then, use this offset position as the target position and perform a clockwise addressing operation on the node hash ring. Starting from the offset position, the search proceeds along the increasing direction of the hash ring value to find the first node position identifier that is greater than or equal to the offset position. If the node is not found even after reaching the end of the ring, the search continues from the starting position. The nodes corresponding to the node position identifiers obtained by each clockwise addressing are added to the candidate set one by one. When the counting traversal is completed, a sequence consisting of multiple node position identifiers is obtained. Each element in this sequence is a node determined by the above offset and addressing operations. This candidate replica node list is output, containing multiple alternative nodes for storing data replicas in addition to the primary node. These nodes are evenly distributed on the hash ring, providing a set of nodes to choose from for subsequent game theory replica number negotiation.

[0042] S44. Based on the candidate replica node list, a cost function is obtained through linear regression cost processing. In this step, each node in the candidate replica node list is traversed first to obtain the preset unit storage price parameter and scarcity penalty coefficient parameter of the node. At the same time, the current remaining storage space capacity of the node is read. For any candidate node, considering the size of the data to be stored, the unit storage price is multiplied by the size of the data to be stored to obtain the basic storage cost. At the same time, the size of the data to be stored is divided by the remaining space capacity of the node to obtain a ratio reflecting the scarcity of space. Then, the scarcity penalty coefficient is multiplied by this ratio to obtain the scarcity penalty cost. Subsequently, the basic storage cost and the scarcity penalty cost are added to obtain the total cost of storing a specified size of data at the node. The above cost calculation relationship is established for each candidate node to form a set of linear function expressions with the size of the stored data as the independent variable and the storage cost as the dependent variable, which is the cost function.

[0043] S45. Data availability benefit processing based on cost function to generate benefit function; In this step, firstly, the availability probability value of each node in the candidate replica node list is obtained. This value is between zero and one, representing the possibility that the node can provide data access service under normal operating conditions. For any selected number of replicas, the availability probabilities of several nodes corresponding to that number are inverted, that is, one is subtracted from the availability probability of each node to obtain the data unavailability probability of each node. Then, the unavailability probabilities of all selected nodes are multiplied together, that is, the unavailability probability values ​​of all nodes are multiplied sequentially to obtain a comprehensive product result. This result represents the joint probability of all replica nodes failing at the same time. Then, one is subtracted from the product to obtain the overall data availability benefit value under the replica number configuration. This benefit value gradually increases with the increase of the number of replicas, but the growth rate shows a decreasing trend. With the number of replicas as the independent variable, the functional relationship between data availability benefit and the number of replicas is constructed using the above calculation process to form the benefit function. This benefit function is the output, which quantifies the data reliability improvement effect brought by different replica number configurations in the form of a mathematical expression.

[0044] S46. Perform particle swarm optimization on the revenue function to generate the globally optimal data storage state. In this step, the revenue function and the cost function of each candidate node are received first. At the same time, the preset cost weight coefficient and the maximum number of replicas are obtained. The number of replicas is used as the decision variable to be optimized. Its value range is an integer between 1 and the maximum number of replicas. For each possible number of replicas, the corresponding number of nodes are selected from the candidate replica node list according to the number. The data availability revenue value corresponding to the number is calculated by the revenue function. At the same time, the cost function of the selected nodes is called to calculate the cost of storing the undetermined amount of data on each node. The costs of all nodes are summed and then the sum is divided by the number of replicas to obtain the average storage cost. The average storage cost is multiplied by the cost weight coefficient to obtain the weighted cost value. Subsequently, the weighted cost value is subtracted from the data availability benefit value to obtain the net benefit value for that number of replicas. The net benefit values ​​corresponding to all possible replica numbers are calculated sequentially. By comparison, the number of replicas corresponding to the maximum net benefit is found and determined as the optimal number of replicas for this round of negotiation. Based on the optimal number of replicas, corresponding nodes are selected, and combined with the storage cost and availability probability of each node, a globally optimized data storage state containing the optimal number of replicas, a specific node list, and the expected net benefit is generated. This globally optimized data storage state serves as the output, achieving a Nash equilibrium between data availability benefit and storage cost. This enables the intelligent data storage method based on edge computing and disk collaboration to meet data reliability requirements in a way that minimizes the overall cost. Based on the globally optimized data storage state, the optimal number of replicas, the node list, and the satisfaction of availability requirements can be achieved while minimizing storage costs can be minimized.

[0045] S5. Based on the globally optimized data storage state, the data is dynamically partitioned and stored to obtain the final hard disk data storage state. In this step, the globally optimized data storage state obtained through multi-node collaborative optimization is used as input. The physical hard disk is dynamically partitioned and allocated locally on the edge node according to the input. For example, the high-frequency zone and low-frequency zone are divided according to the data popularity or the partition boundary is adjusted according to the node load balance. The actual data is written to the disk, and finally a persistent hard disk data storage state that meets the global optimization goal is formed.

[0046] This invention uses consistent hashing to map nodes and data blocks to a ring space, generating a uniformly distributed list of candidate replica nodes, providing a structured and scalable distributed foundation for replica placement. In a smart city edge node cluster, some nodes located in the core computer room have an availability of up to 99.99%, but the system still forces the storage of three replicas, resulting in about 30% of the storage space being occupied by redundant data. More valuable historical monitoring data that could have been stored was forced to be cleared in advance due to insufficient space. The decision on the number of replicas is modeled as a game theory problem. The data reliability benefits under different numbers of replicas are calculated through the exponential decay availability model. For example, an edge node has occupied 90% of the storage capacity due to its core business, but consistent hashing still selects it as the replica node for new data, causing the node to quickly fill up and trigger write rate limiting, while other idle nodes are not fully utilized due to hash position. By solving the Nash equilibrium point using the particle swarm optimization algorithm, the global storage cost is minimized while ensuring availability, effectively solving the problem of excessive data redundancy between edge nodes. For example, in an edge computing scenario in an offshore oil field, some nodes are deployed on unmanned platforms and rely on satellite communication to synchronize with the shore. The stability of the communication link changes drastically with the weather. The existing redundancy strategy with a fixed number of replicas cannot detect this dynamic change. When the probability of communication interruption increases during the typhoon season, the system still maintains a two-replica configuration. Once the master node and the replica node go offline due to the link interruption, the critical production data will be unrecoverable. However, when the wind and waves are calm, the communication link is highly reliable, and a two-replica configuration is sufficient. The system cannot dynamically reduce the number of replicas to free up storage space.

[0047] Example 2: Current technologies lack consideration for cost factors such as remaining node space and storage unit price, resulting in low global storage utilization. This leads to unnecessary redundancy and waste when node reliability is high, while failing to provide sufficient fault tolerance when node reliability is insufficient. Please refer to [link / reference needed]. Figure 2 The diagram shown is a structural block diagram of a data storage system based on edge computing provided in this embodiment. The system includes a preprocessing module, a hash data indexing module, a prefetch cache module, a global optimization storage module, and a dynamic storage module. The preprocessing module is used to acquire the raw data stream from the target data device and obtain preprocessed data blocks based on the raw data stream through data preprocessing. The hash data index module is used to predict the heat load based on preprocessed data blocks, generate write pressure prediction values, perform threshold clustering location lookup on the write pressure prediction values ​​to obtain the target storage area, and generate an optimized write data index based on the target storage area through health index construction. The prefetch cache module is used to embed behavioral sequence features into the optimized write data index, generate data block embedding vectors, perform co-sequence processing based on the data block embedding vectors to generate temporal feature vectors, and process the temporal feature vectors through the attention weight candidate set to generate prefetch cache data. The global optimal storage module is used to obtain the master node identifier by processing the node affiliation of the prefetched cache data, perform virtual regression cost processing based on the master node identifier to obtain the cost function, perform availability balancing processing on the cost function, and generate the global optimal data storage status. The dynamic storage module is used to dynamically partition and store data based on the globally preferred data storage state to obtain the final hard disk data storage state.

[0048] Example 3: Existing technologies lack consideration for cost factors such as remaining node space and storage unit price, resulting in low global storage utilization. This leads to unnecessary redundancy and waste when node reliability is high, and fails to provide sufficient fault tolerance when node reliability is insufficient. This embodiment also provides a computer-readable storage medium storing computer program instructions, which, when executed, implement the steps of any of the edge computing-based data storage methods mentioned above.

[0049] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code, including but not limited to disk storage, CD-ROM, optical storage, etc.

[0050] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data storage method based on edge computing, characterized in that, The steps of this method are as follows: acquire the raw data stream of the target data device, and obtain preprocessed data blocks by preprocessing the raw data stream; Based on the preprocessed data blocks, the heat load is predicted to generate write pressure prediction values. Threshold clustering is performed on the write pressure prediction values ​​to find the target storage area. Based on the target storage area, the health index is used to build and process the data to generate an optimized write data index. Behavioral sequence features are embedded into the optimized write data index to generate data block embedding vectors. Cooperative sequence processing is performed based on the data block embedding vectors to generate temporal feature vectors. The temporal feature vectors are then processed through an attention weight candidate set to generate prefetch cache data. Based on the prefetched cache data, the master node identifier is obtained through node affiliation processing. Virtual regression cost processing is performed based on the master node identifier to obtain the cost function. Availability balancing processing is then performed on the cost function to generate the globally optimal data storage state. The final hard disk data storage state is obtained by dynamically partitioning and storing data based on the globally optimized data storage state.

2. The data storage method based on edge computing according to claim 1, characterized in that, Based on the raw data stream, data preprocessing is performed, including: extracting structured data frames from the raw data stream through protocol parsing; Sliding outlier processing is performed based on structured data frames to generate effective data payloads; The effective data payload is divided into preprocessed data blocks by a fixed duration.

3. The data storage method based on edge computing according to claim 1, characterized in that, Based on the preprocessed data blocks, heat load prediction is performed to generate write pressure prediction values. Threshold clustering location lookup is performed on the write pressure prediction values, including: processing the preprocessed data blocks through data block heat labeling to obtain heat label data blocks. Short-term load prediction is performed based on heat tag data blocks to generate write pressure prediction values; Threshold clustering is performed on the predicted pressure values ​​to generate pressure level codes. The target storage area is obtained by performing a hash location lookup based on the pressure level code.

4. The data storage method based on edge computing according to claim 1, characterized in that, The process involves building a health index based on the target storage region, including: writing throttling based on the health index of the target storage region to obtain optimized write instructions; The optimized write commands are indexed to generate an optimized write data index.

5. The data storage method based on edge computing according to claim 1, characterized in that, The optimized write data index is embedded with behavioral sequence features to generate a data block embedding vector. Based on the data block embedding vector, collaborative sequence processing is performed, including: constructing a user access vector based on the optimized write data index through user behavior sequences. Data block feature embedding is performed based on user access vectors to generate data block embedding vectors; Perform co-similarity processing on the data block embedding vectors to obtain a list of similar data blocks; A time-series feature vector is generated by accessing the sequence based on a list of similar data blocks.

6. The data storage method based on edge computing according to claim 1, characterized in that, The attention weight candidate set is processed based on the temporal feature vector, including: attention weight allocation based on the temporal feature vector to obtain the attention weight vector; The attention weight vector is prefetched to generate prefetch cache data.

7. The data storage method based on edge computing according to claim 1, characterized in that, Based on the prefetched cache data, the master node identifier is obtained through node attribution processing. Virtual regression cost processing is then performed based on the master node identifier, including: obtaining the node hash ring through node mapping processing based on the prefetched cache data. Data attribution is processed based on node hash rings to obtain the master node identifier; The master node identifier is virtually mapped to generate a list of candidate replica nodes; The cost function is obtained by linear regression cost processing based on the candidate replica node list.

8. The data storage method based on edge computing according to claim 1, characterized in that, The cost function is processed for availability balancing, including: processing data availability benefits based on the cost function to generate a benefit function; The payoff function is subjected to particle swarm equilibrium processing to generate the globally optimal data storage state.

9. A system applied to the edge computing-based data storage method according to any one of claims 1-8, characterized in that, The system includes: The preprocessing module is used to acquire the raw data stream from the target data device and obtain preprocessed data blocks based on the raw data stream through data preprocessing. The hash data index module is used to predict the heat load based on preprocessed data blocks, generate write pressure prediction values, perform threshold clustering location lookup on the write pressure prediction values ​​to obtain the target storage area, and generate an optimized write data index based on the target storage area through health index construction. The prefetch cache module is used to embed behavioral sequence features into the optimized write data index, generate data block embedding vectors, perform collaborative sequence processing based on the data block embedding vectors to generate temporal feature vectors, and process the temporal feature vectors through the attention weight candidate set to generate prefetch cache data. The global optimal storage module is used to obtain the master node identifier by processing the node affiliation of the prefetched cache data, perform virtual regression cost processing based on the master node identifier to obtain the cost function, perform availability balancing processing on the cost function, and generate the global optimal data storage status. The dynamic storage module is used to dynamically partition and store data based on the globally preferred data storage state to obtain the final hard disk data storage state.

10. A computer-readable storage medium having stored thereon computer program instructions that, when executed, implement the steps of the edge computing-based data storage method according to any one of claims 1 to 8.

Images