Data storage method and device, equipment, storage medium and program product

Abstract

The invention discloses a data storage method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining data features and data attributes of to-be-stored data when the to-be-stored data is received; obtaining node information of all nodes in the resource center, and screening out candidate nodes from all the nodes according to the node information; according to the data features and the node information of the candidate nodes, target nodes matched with the to-be-stored data are screened out; determining a target storage hierarchy corresponding to the target node according to the data attribute; and storing the to-be-stored data to the target storage hierarchy. By adopting the embodiment of the invention, the dynamic scheduling of data storage can be realized, the problem of non-uniform node load of the traditional centralized architecture is effectively solved, and the resource utilization rate of the node is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data storage method, apparatus, device, storage medium, and program product. Background Technology

[0002] Most existing data storage systems still employ a traditional centralized architecture, the core characteristic of which is that all data uploads, storage, and distribution heavily rely on a single central node. All types of data must be directly aggregated to the central node for reception and storage, and all data requests must be uniformly responded to, processed, and forwarded by the central node. Furthermore, this centralized architecture typically uses fixed node allocation strategies, such as round-robin or hash algorithms. The drawback of these strategies is the lack of dynamic awareness of the real-time operating status of nodes and the characteristics of the data itself. This can lead to situations where the central node's CPU utilization is already at a high load, yet the system continues to assign new upload or processing tasks to it, or forcibly writes large files unsuitable for edge node storage to the edge cache, ignoring the storage capacity and access characteristics limitations of the edge nodes. This static and insensitive allocation model directly causes a severe imbalance in load distribution among nodes, resulting in some nodes operating under overload for extended periods while others remain idle, reducing overall load balancing. Summary of the Invention

[0003] The purpose of this invention is to provide a data storage method, apparatus, device, storage medium, and program product that can realize dynamic scheduling of data storage, effectively solve the problem of uneven node load in traditional centralized architecture, and improve the resource utilization of nodes.

[0004] To achieve the above objectives, embodiments of the present invention provide a data storage method, comprising: Upon receiving data to be stored, the data characteristics and data attributes of the data to be stored are obtained; Obtain node information for all nodes in the resource center, and filter candidate nodes from all nodes based on the node information; Based on the data features and the node information of the candidate nodes, target nodes that match the data to be stored are selected. Determine the target storage level corresponding to the target node based on the data attributes; The data to be stored is stored in the target storage level.

[0005] As an improvement to the above solution, obtaining the data features and data attributes of the data to be stored includes: The data to be stored is subjected to multimodal feature extraction using a data extraction model to obtain data features; The data attributes of the data to be stored are determined based on the data characteristics.

[0006] As an improvement to the above scheme, the data characteristics include at least one of data size, access frequency, and data type.

[0007] As an improvement to the above scheme, the data attributes include high-frequency access data, medium-frequency access data, and low-frequency access data; wherein, the high-frequency access data is data whose access frequency is in a first interval, the medium-frequency access data is data whose access frequency is in a second interval, and the low-frequency access data is data whose access frequency is in a third interval, the lower limit of the access frequency in the first interval is higher than the upper limit of the access frequency in the second interval, and the lower limit of the access frequency in the second interval is higher than the upper limit of the access frequency in the third interval.

[0008] As an improvement to the above scheme, the data extraction model is a hybrid data extraction model composed of a Transformer model and a first neural network model; wherein, the Transformer model is used to extract data features of text data, and the first neural network model is used to extract data features of at least one type of data, including image data, video data, and audio data.

[0009] As an improvement to the above scheme, the node information includes at least one of the following: node operating status, resource supply data, and deployment information.

[0010] As an improvement to the above scheme, the step of filtering candidate nodes from all nodes based on the node information includes: Based on the network topology relationships and node information among all nodes, construct a node state graph; The node state graph is input into the second neural network model, and the scheduling priority matrix output by the second neural network model is obtained; All nodes are sorted based on the scheduling priority matrix, and the top n nodes in the priority sort are selected as candidate nodes, where n is an integer and n≥2.

[0011] As an improvement to the above scheme, the step of filtering out target nodes that match the data to be stored based on the data features and the node information of the candidate nodes includes: Calculate the matching degree between the data features and the node information of each candidate node; The candidate node with the highest matching degree is selected as the target node for the data to be stored.

[0012] As an improvement to the above solution, determining the target storage level corresponding to the target node based on the data attributes includes: If the data attributes are high-frequency access data and medium-frequency access data, the SSD storage layer is determined as the target storage layer corresponding to the target node; If the data attribute is low-frequency access data, the object storage layer is determined as the target storage layer corresponding to the target node.

[0013] As an improvement to the above solution, after determining the target storage level corresponding to the target node based on the data attributes, the method further includes: Invoke the storage optimization model; The storage optimization model is solved by the simulated annealing algorithm to obtain the capacity allocation scheme of the storage hierarchy; The capacity of the storage tier is adjusted based on the capacity allocation scheme.

[0014] As an improvement to the above scheme, the storage optimization model is constrained by the access latency not exceeding a preset latency upper limit, and the optimization objective is to minimize the total storage cost.

[0015] As an improvement to the above scheme, the total storage cost is obtained by summing the single-level storage cost of each storage tier, wherein the single-level storage cost is the product of the unit storage cost and the storage capacity corresponding to the current storage tier.

[0016] As an improvement to the above solution, after storing the data to be stored in the target storage level, the method further includes: Obtain the vector clock of each stored data in each storage level, and use the vector clock to determine the change information of the stored data; A synchronization queue model is constructed based on the data attributes and change frequency of the stored data, and the synchronization queue model is used to generate the scheduling order for data synchronization. Based on the scheduling order, the stored data is synchronized.

[0017] As an improvement to the above scheme, the step of using the vector clock to determine the change information of the stored data includes: Compare the vector clocks of the same stored data on different nodes; If at least two nodes have mismatched vector clocks, it is determined that the stored data has undergone an incremental change; Obtain incremental data of stored data that has undergone incremental changes, and perform a hash operation on the incremental data to generate hash fingerprint data; The change information is determined based on the hash fingerprint data.

[0018] As an improvement to the above scheme, the synchronization queue model is a correlation model between state and action value; wherein, the state includes the data attribute and the change frequency, the action value corresponds to different levels of synchronization priority, and the synchronization queue model aims to maximize the expected return of the correlation function between the state and the action value to generate the scheduling order of data synchronization.

[0019] As an improvement to the above solution, the step of synchronizing the stored data includes: The node containing the stored data that has not undergone incremental changes is used as the synchronization node; The change information is sent to the synchronization node.

[0020] As an improvement to the above scheme, the resource center includes a central node and at least one edge node associated with the central node.

[0021] As an improvement to the above solution, storing the data to be stored in the target storage level includes: When the target storage level is located at the central node, the data to be stored is sent to the target storage level corresponding to the central node; When the target storage level is located at the edge node, the data to be stored is sent to the central node, and then the central node synchronizes the data to be stored to the target storage level corresponding to the edge node.

[0022] As an improvement to the above scheme, after storing the data to be stored, or after synchronizing the data to be stored to the target storage level corresponding to the edge node, the central node sends a notification message to the media plane associated with the data to be stored, so that the media plane can retrieve the data from the corresponding storage level.

[0023] As an improvement to the above solution, when receiving data to be stored, the method further includes: The data to be stored is subject to content review.

[0024] To achieve the above objectives, embodiments of the present invention also provide a data storage device, comprising: The data acquisition module is used to acquire the data characteristics and data attributes of the data to be stored when it receives the data to be stored. The node information acquisition module is used to acquire node information of all nodes in the resource center; The candidate node filtering module is used to filter candidate nodes from all nodes based on the node information. The target node filtering module is used to filter out target nodes that match the data to be stored based on the data features and the node information of the candidate nodes. A storage level determination module is used to determine the target storage level corresponding to the target node based on the data attributes. The data storage module stores the data to be stored in the target storage level.

[0025] To achieve the above objectives, embodiments of the present invention also provide a data storage device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the data storage method as described in any of the above embodiments.

[0026] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the data storage method as described in any of the above embodiments.

[0027] To achieve the above objectives, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the data storage method as described in any of the above embodiments.

[0028] Compared to existing technologies, the data storage method, apparatus, device, storage medium, and program products disclosed in this invention, based on a two-way matching mechanism of data characteristics and node information, can accurately select target nodes suitable for storing data, avoiding the problem of low storage efficiency caused by resource mismatch and improving node resource utilization. Furthermore, through precise mapping of data attributes to the storage level of target nodes, a differentiated storage strategy is achieved: high-speed storage of frequently accessed data and low-cost archiving of less frequently accessed data. This satisfies the low-latency access requirements of frequently accessed data while effectively reducing overall storage costs. Simultaneously, the hierarchical decision-making logic of candidate node selection and target node matching ensures the flexibility and reliability of storage deployment. Adapting to the collaborative storage architecture of central and edge nodes in a distributed resource center, it enables dynamic scheduling of data storage, effectively solving the problem of uneven node load in traditional centralized architectures, improving node resource utilization, and is suitable for business scenarios such as 5G new calls that have high requirements for balancing storage performance and cost. Attached Figure Description

[0029] Figure 1 This is a structural block diagram of the data storage system provided in an embodiment of the present invention; Figure 2 This is a flowchart of a data storage method provided in an embodiment of the present invention; Figure 3 This is a flowchart of the process for filtering candidate nodes provided in an embodiment of the present invention; Figure 4This is a schematic diagram of the process of the central node notifying the media plane to pull data, provided in an embodiment of the present invention; Figure 5 This is a flowchart of data synchronization between nodes provided in an embodiment of the present invention; Figure 6 This is a structural block diagram of a data storage device provided in an embodiment of the present invention; Figure 7 This is a structural block diagram of a data storage device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] See Figure 1 , Figure 1 This is a structural block diagram of the data storage system provided in this embodiment of the invention. The system includes a third-party platform, a business gateway, and a data processing system (including a core processing module, an auditing module, and a media resource module). These modules work together to perform permission verification, security auditing, intelligent storage, and resource synchronization of the data to be stored. The third-party platform, acting as the data uploader, sends the data to be stored to the business gateway. Upon receiving the data, the business gateway uses a Redis cache token to complete permission verification. After successful verification, the data is forwarded to the core processing module of the data processing system. The core processing module parses the data request and sends it to the auditing module. The auditing module performs malicious attack protection through protocol parsing and other methods. For example, the auditing module pushes the data to a content auditing platform. The platform uses the ResNet50 deep learning algorithm for content review and optimizes the model through L2 regularization to avoid overfitting and ensure auditing accuracy. Based on the content auditing results, if the data complies with the rules, the core processing module sends it to the media resource module for storage. The core processing module can simultaneously perform data processing, node filtering, and storage level confirmation, thereby storing the data to be stored in that storage level. This system achieves end-to-end verification of permissions, security, and content through multi-module layered collaboration. At the same time, it combines the storage and synchronization mechanism of distributed nodes to balance the security and compliance of data storage with the efficiency of business calls.

[0032] See Figure 2 , Figure 2 This is a flowchart of a data storage method provided in an embodiment of the present invention. The data storage method is applied to the data processing system and includes steps S1 to S5.

[0033] S1. When receiving data to be stored, obtain the data characteristics and data attributes of the data to be stored.

[0034] For example, the data to be stored consists of various resource data sent by third-party platforms based on business needs. These third-party platforms may be 5G new voice service providers, video conferencing platforms, online education platforms, or enterprise cloud collaboration platforms. The data to be stored includes voice clips, video ringback tones, and real-time captions in new voice scenarios; meeting recordings, shared documents, and participant avatars in video conferencing scenarios; and course videos, courseware PPTs, and exercise audio in online education scenarios. The third-party platform triggers data uploads immediately after a user completes a specific business operation, or triggers uploads in batches based on a platform-configured timed strategy, or triggers uploads when the data meets preset threshold conditions. The data characteristics are used to indicate the objective physical attributes and business access characteristics of the data to be stored, providing a quantitative basis for subsequent node adaptation and storage strategy selection. The data attributes are used to indicate the business value level of the data to be stored, providing a classification standard for accurate matching of storage levels.

[0035] Furthermore, when receiving data to be stored, the method further includes: performing content review on the data to be stored.

[0036] For example, after receiving data to be stored uploaded by a third-party platform, the business gateway first performs preliminary security screening through protocol parsing and format verification to identify and block malicious data that does not conform to transmission protocol specifications or has abnormal file formats, such as abnormal programs disguised as video files. Subsequently, the core processing module pushes the data to the professional content review module, which uses the ResNet50 deep learning algorithm to conduct in-depth review of the multimodal data. For example, for text data, it detects whether it contains sensitive words or illegal expressions; for image / video data, it identifies whether there are illegal scenes or inappropriate information; and for audio data, it detects whether it contains illegal voice content. Simultaneously, to avoid overfitting of the algorithm model leading to a decrease in review accuracy, L2 regularization is used to optimize model parameters, suppressing redundant parameters while retaining core features to ensure the accuracy and stability of the review results. Only data confirmed as compliant through the above multi-layered review process enters the subsequent data feature extraction and storage node matching process. If illegal data is found during the review, it is directly intercepted, and a review failure response is returned to the third-party platform, preventing illegal data from entering the storage system at the source.

[0037] Furthermore, the data characteristics include at least one of data size, access frequency, and data type.

[0038] For example, data size refers to the actual storage space occupied by the data to be stored. For instance, a voice clip from a 5G call might be a few MB, while a high-definition online education course video might be tens of GB. This directly determines the capacity adaptation requirements of storage nodes. Access frequency refers to the number of times the data to be stored is accessed by business systems or users within a preset time period. For example, popular video ringback tones can be accessed tens of thousands of times per day, while historical meeting recordings might be accessed less than 10 times per month. This is a core basis for classifying data attributes. Data type refers to the format and carrier category of the data to be stored. Based on multimodal scenarios, it can be divided into text, image, video, and audio types, etc. Different types of data have different storage formats and read / write performance requirements, directly affecting the adaptation selection of storage levels and nodes.

[0039] In this embodiment of the invention, from the perspective of data size, the resource supply capacity of nodes can be matched. For example, large files are prioritized for scheduling to nodes with sufficient storage capacity, avoiding service interruptions caused by storage limits on small-capacity nodes. From the perspective of access frequency, the attribute classification of hot, warm, and low-frequency accessed data can be directly supported. For example, high-frequency data corresponds to low-latency storage levels, ensuring rapid access to high-frequency business resources. From the perspective of data type, the storage specifications of different types of data can be adapted. For example, video data is matched with high-bandwidth nodes, and text data is adapted to lightweight storage media, thereby improving the accuracy of storage resource adaptation. When these three aspects are combined, the one-sidedness of single-feature decision-making is avoided, and the selection of storage nodes and storage levels is more in line with the actual business attributes of the data. Ultimately, this achieves efficient allocation of storage resources, optimization of business response speed, and reasonable control of storage costs.

[0040] Furthermore, the data attributes include high-frequency access data, medium-frequency access data, and low-frequency access data; wherein, the high-frequency access data is data whose access frequency is in a first interval, the medium-frequency access data is data whose access frequency is in a second interval, and the low-frequency access data is data whose access frequency is in a third interval, wherein the lower limit of the access frequency in the first interval is higher than the upper limit of the access frequency in the second interval, and the lower limit of the access frequency in the second interval is higher than the upper limit of the access frequency in the third interval.

[0041] For example, high-frequency access data refers to high-value data with the highest access frequency within a preset period (e.g., ≥1000 accesses per day), such as popular video ringback tones on 5G new calling platforms and real-time shared documents on video conferencing platforms. Mid-frequency access data refers to regular data with an access frequency in the middle range (e.g., 10-1000 accesses per day), such as recently released online education course videos and office documents on enterprise cloud collaboration platforms. Low-frequency access data refers to low-frequency access data with the lowest access frequency (e.g., <10 accesses per day), such as historical meeting recordings and archived courseware. This three-level classification allows for matching the optimal storage strategy to data with different access needs, achieving a dynamic balance between performance and cost.

[0042] In this embodiment of the invention, by clearly distinguishing the levels through access frequency ranges, differentiated and precise matching of storage resources can be achieved. This classification not only avoids the waste of resources caused by a one-size-fits-all approach to storage, but also allows data with different access needs to be matched with the optimal storage strategy, effectively improving storage efficiency. At the same time, it is compatible with the hierarchical architecture of distributed nodes and supports flexible business expansion.

[0043] Further, step S1 specifically includes: using a data extraction model to extract multimodal features from the data to be stored to obtain data features; determining the data attributes of the data to be stored based on the data features; wherein, the data extraction model is a hybrid data extraction model composed of a Transformer model and a first neural network model; wherein, the Transformer model is used to extract data features from text data, and the first neural network model is used to extract data features from at least one of image data, video data, and audio data.

[0044] For example, the first neural network model is a CNN (Convolutional Neural Network), while the Transformer model is a sequence modeling model based on the self-attention mechanism in deep learning. For various types of data uploaded from third-party platforms, a hybrid model consisting of the Transformer model and the first neural network model is activated. For text data, the self-attention mechanism of the Transformer model is used to accurately capture core features such as semantic relationships and keyword distribution, including semantic logic in subtitles and core information in documents. For non-text data such as images, videos, and audio, the first neural network model extracts specific features. Image data focuses on color distribution and texture features; video data focuses on inter-frame motion features and keyframe information; and audio data extracts spectral features and voiceprint features. After the features of various data types are extracted, the data to be stored is determined to be high-frequency, medium-frequency, or low-frequency access data according to a preset access frequency range threshold.

[0045] In this embodiment of the invention, a multimodal data extraction model accurately captures the multidimensional features of the data to be stored, overcoming the limitations of single-modal extraction and ensuring the comprehensiveness and accuracy of data features. Furthermore, the hybrid data extraction model avoids the limitations of a single model in adapting to multimodal data while ensuring the integrity and accuracy of features for each type of data. This provides a reliable foundation for subsequent data attribute determination and storage strategy matching, supporting the efficient classification and adaptive storage of multimodal data in a distributed storage system and improving the system's ability to process complex data.

[0046] S2. Obtain the node information of all nodes in the resource center, and filter out candidate nodes from all nodes based on the node information.

[0047] For example, a resource center refers to a core cluster of a distributed storage architecture that provides data storage and resource scheduling services to third-party platforms. It integrates data storage, node management, and resource monitoring, enabling it to coordinate the storage resources of each node and adapt to massive data storage needs across multiple scenarios. The resource center includes a central node and at least one edge node associated with it. The central node is the core control node of the resource center, undertaking core functions such as global resource scheduling, data consistency verification, storage strategy decision-making, and edge node management. It is typically deployed in a core data center, possessing strong computing power, storage capacity, and bandwidth resources, and can aggregate and analyze information from all nodes. Edge nodes are distributed nodes deployed close to users or business terminals, such as regional edge data centers and base station-supporting storage nodes. They directly address the storage and access needs of local businesses, offering low-latency response advantages. They are primarily responsible for storing, caching, and quickly accessing local hot data, while also accepting unified control and scheduling from the central node. The node information is used to indicate the resource status, deployment attributes, and / or service capabilities of each node, providing accurate criteria for candidate node selection.

[0048] Furthermore, the node information includes at least one of the following: node operating status, resource supply data, and deployment information.

[0049] For example, operational status refers to the current service availability and performance of a node, encompassing at least one of real-time metrics such as CPU utilization, memory usage, bandwidth load, and network RTT (Round-Trip Time), used to determine whether the node is in a stable and serviceable state. Resource provisioning data refers to the node's allocable hardware resource reserves, including at least one of remaining storage capacity, idle computing resources, and available bandwidth, serving as the basis for assessing the node's ability to support the data to be stored. Deployment information refers to the node's physical deployment attributes, including its geographical location, providing a reference for node location selection and low-latency access for data storage.

[0050] Further, see Figure 3 , Figure 3 This is a flowchart of the candidate node selection process provided in the embodiment of the present invention, wherein step S2 specifically includes steps S21 to S23.

[0051] S21. Construct a node state graph based on the network topology relationships and node information among all nodes.

[0052] For example, the network topology relationship between the central node and all edge nodes within the resource center is clearly defined. Nodes are the vertices of the graph, and network connections between nodes (such as communication links and data transmission channels) are the edges. The weight of the edges can be quantified by network indicators such as RTT and bandwidth utilization between nodes. For instance, the smaller the RTT, the higher the edge weight, indicating higher communication efficiency between nodes. Secondly, core node information such as the operating status, resource supply data, and deployment information of each node is associated with the corresponding nodes as vertex attributes. This ultimately forms a node state graph with nodes as the core, network connections as the associations, and node status as the attributes. This graph can comprehensively characterize the capabilities of each node and the collaborative potential between nodes, providing multi-dimensional input data for subsequent priority calculations.

[0053] S22. Input the node state graph into the second neural network model and obtain the scheduling priority matrix output by the second neural network model.

[0054] For example, the second neural network model is a GNN (Graph Neural Network), which uses inference to quantitatively evaluate node scheduling priorities. The second neural network model, taking advantage of the "vertex-edge-attribute" structure of the node state graph, captures both the individual node's state features (such as sufficient remaining storage capacity) and inter-node relationship features (such as communication efficiency between edge and center nodes) through graph convolution operations or attention mechanisms. The GNN model fuses and encodes these features, transforming them into high-dimensional feature vectors, which are then mapped to the scheduling priority space through fully connected layers, ultimately outputting a two-dimensional scheduling priority matrix that satisfies: N is the number of nodes. The row index corresponds to each node in the resource center, and the column index corresponds to the storage requirement dimension of the data to be stored, such as capacity adaptation requirement, low latency requirement, stability requirement, etc. The element value in the matrix represents the priority score of the corresponding node in meeting the requirement of that dimension. The higher the score, the stronger the adaptation, thus realizing the precise matching and quantification of nodes and storage requirements.

[0055] For example, an embodiment of the present invention provides a mathematical expression for scheduling priority of GNN computing nodes: (1); in, This represents the scheduling priority score from node i to node j, i.e., the degree of fit when node i schedules data to be stored to node j. Activation functions, such as Sigmoid and ReLU, are used to map the original feature values ​​of the GNN output to a reasonable priority range, such as 0 to 1, to facilitate quantization and comparison. This represents the load data of node i, corresponding to the load of CPU, memory, or storage, reflecting the current resource usage status of node i. This represents the bandwidth and network status between node i and node j, reflecting the communication efficiency between nodes; This represents the storage resource data of node j, such as the remaining storage capacity, and represents the storage carrying capacity of node j.

[0056] It should be noted that the GNN model takes the load of node i, the network states of nodes i and j, and the storage resources of node j as input. It extracts features through graph structure modeling (considering both the node's own state and the relationships between nodes), and after transformation by an activation function, outputs a priority score for scheduling data from node i to node j. Finally, it calculates the priority score for all node pairs. It will form a scheduling priority matrix This provides a quantitative basis for subsequent candidate node selection. In addition, only some node information appears in formula (1), which is an exemplary form for simplification. In actual application, all the information contained in the node state map can be used as the input dimension of GNN to more comprehensively characterize the comprehensive ability and adaptability of the node. Therefore, the form of formula (1) is not limited to this.

[0057] S23. Sort all nodes based on the scheduling priority matrix, and select the top n nodes in the priority sort as candidate nodes, where n is an integer and n≥2.

[0058] For example, node ranking and candidate selection are achieved through the quantitative scores of the priority matrix, ensuring the quality and redundancy backup capabilities of the candidate nodes. For instance, based on the node scores in the scheduling priority matrix, all nodes are ranked from highest to lowest. Nodes with higher scores represent those whose hardware resources, network performance, and deployment location are more suitable for the storage requirements of the data to be stored. The top n nodes in the ranking are selected as candidate nodes, and the specific value of n can be configured according to the total number of nodes in the resource center and the business reliability requirements.

[0059] In this embodiment of the invention, a node state graph is constructed by integrating network topology and node information, comprehensively integrating node deployment logic and resource status. Furthermore, a scheduling priority matrix is ​​generated using a second neural network model to achieve intelligent quantitative evaluation of node adaptability. Additionally, selecting the top n candidate nodes based on matrix sorting avoids the randomness and limitations of single-node selection while ensuring the resource adaptability and reliability of candidate nodes. This provides a high-quality candidate pool for accurate matching of subsequent target nodes, improving the node scheduling efficiency and storage deployment flexibility of the distributed storage system.

[0060] S3. Based on the data features and the node information of the candidate nodes, select the target nodes that match the data to be stored.

[0061] For example, candidate nodes are selected from all nodes that have sufficient resources and stable status. However, the capabilities of different candidate nodes may not be a perfect match for the needs of the data to be stored. Therefore, it is necessary to further filter by combining the data characteristics of the data to be stored (the actual needs of the data) and the node information (the actual capabilities of the node) to avoid situations where resources meet the requirements but the needs do not match, thereby ensuring that the target node can accurately match the business scenario and performance requirements of the data.

[0062] Further, step S3 specifically includes: calculating the matching degree between the data features and the node information of each candidate node; and selecting the candidate node with the highest matching degree as the target node of the data to be stored.

[0063] For example, from the data features obtained in step S1, a resource feature vector F=[Size,Frequency,Type] is extracted, where Size is the data size, Frequency is the access frequency, and Type is the data type. The resource feature vector represents the core requirements of the data to be stored. From the node information of the candidate nodes, a node capability label C=[CacheCap,StorageType,GeoZone] is extracted, where CacheCap is the cache capacity, StorageType is the storage type, and GeoZone is the geographical region. The node capability label represents the service capability of the candidate node. Cosine similarity is used to quantify the matching degree between data features and node capabilities. Simultaneously, the matching result is corrected by combining the network RTT between nodes. This process satisfies the following formula: (2); in, A matching score is used to characterize the degree of matching; the higher the score, the better the match. It is the cosine similarity between resource characteristics and node capabilities. The higher the value, the better the data requirements match the node capabilities. It is a normalized value of the network latency between nodes, reflecting the impact of access latency on the matching degree; , These are dynamic weights trained from historical scheduling data, and their weight ratios can be adjusted according to business scenarios (such as low latency priority, capacity priority). The calculation of all candidate nodes... Then, the node with the highest matching degree is selected as the target node.

[0064] In this embodiment of the invention, by quantifying the matching degree between data features and candidate node information, the storage requirements and node resources are accurately matched, avoiding resource mismatch caused by subjective selection. The node with the highest matching degree is selected as the target node, ensuring that the data to be stored can be stored by relying on the optimally matched node resources, and reducing the problem of uneven node load.

[0065] S4. Determine the target storage level corresponding to the target node based on the data attributes.

[0066] For example, a target storage tier is a storage resource module within the target node that is adapted to the data attributes. Different data attributes correspond to different storage tiers. Central nodes typically feature a complete multi-layered storage tier, including an SSD (Solid-State Drive) storage layer (such as an SSD cluster) and an object storage layer, such as HDFS (Hadoop Distributed File System). Meanwhile, edge nodes, deployed closer to the business terminals, typically prioritize low-latency storage resources, primarily configuring lightweight SSD storage layers and object storage layers. It should be noted that the above storage tier division is only an example; in practical applications, the storage tier corresponding to a node may not be limited to this.

[0067] Furthermore, step S4 specifically includes the following situations: 1) If the data attributes are high-frequency access data and medium-frequency access data, the SSD storage layer is determined as the target storage layer corresponding to the target node.

[0068] For example, the core requirements for frequently accessed data are low latency response and high access hit rate. Edge node SSD caches offer read / write speeds in the microsecond to millisecond range and are compatible with the LRU-K (K=3) cache replacement algorithm, prioritizing the retention of frequently accessed data and significantly reducing terminal access waiting time. Simultaneously, the deployment characteristics of edge nodes shorten network transmission paths, further improving response efficiency. For instance, in new 5G call scenarios, 500KB of popular emoji data, after being written as frequently accessed data into the edge node's first-level cache layer, can have its access latency controlled within 10ms, meeting users' real-time access needs.

[0069] For example, mid-frequency access data requires a balance between response speed and storage cost. SSD clusters offer both high performance and moderate storage cost, with read / write latency down to the millisecond level, meeting the response requirements of mid-frequency access scenarios. Simultaneously, consistent hashing algorithms enable load balancing among cluster nodes, preventing overload of a single node and ensuring the stability of data storage and access. For instance, recently released course materials on online education platforms, as mid-frequency access data, are stored in a central SSD cluster, with response latency controlled within 50ms. This satisfies students' daily learning efficiency needs while avoiding the use of high-cost caching resources.

[0070] It should be noted that if the dynamic node scheduling selects the central node, it means that the current state of the central node (low load, sufficient bandwidth) is the best in the full node pool, but the resource is hot data (requiring low latency access). In this case, the storage tiering strategy will select the local SSD cache of the central node instead of the edge node, which still meets the requirements of hot data SSD storage and low latency.

[0071] 2) If the data attribute is low-frequency access data, the object storage layer is determined as the target storage layer corresponding to the target node.

[0072] For example, the core requirement for low-frequency access data is low-cost, high-capacity long-term archiving storage. Object storage layers offer large-capacity storage capabilities and support erasure coding technology. Through data sharding and redundancy verification mechanisms, they can reduce redundancy costs by 50% compared to traditional multi-replica storage while ensuring data reliability, significantly reducing long-term storage overhead. For instance, archived historical meeting recordings in enterprise cloud collaboration platforms and expired course videos in online education platforms, as low-frequency access data, can be migrated to object storage layers. This allows for compliance archiving requirements while minimizing storage resource costs.

[0073] In this embodiment of the invention, the storage hierarchy is precisely determined according to the data attributes to achieve optimal adaptation between the storage hierarchy and data access requirements.

[0074] Furthermore, after performing step S4, the method further includes: invoking the storage optimization model; solving the storage optimization model using the simulated annealing algorithm to obtain a capacity allocation scheme for the storage level; and adjusting the capacity of the storage level based on the capacity allocation scheme.

[0075] For example, the storage optimization model refers to a two-dimensional Pareto optimization model of storage cost and access latency. It is a decision-making model that minimizes the total storage cost under the constraint that the access latency does not exceed the business tolerance threshold. This model uses the capacity of the storage tier as a variable and the unit cost of different storage media as variables. Simulated annealing is a stochastic optimization algorithm based on a Monte Carlo iterative solution strategy. Inspired by the solid annealing process, it simulates the disordered movement of particles at high temperatures and their gradual stabilization during cooling, randomly searching for the optimal solution in the solution space, effectively avoiding getting trapped in local optima. In this embodiment of the invention, this algorithm is used to solve the storage optimization model, using the capacity allocation of different storage tiers as the solution, with the objectives of minimizing storage cost and achieving the required access latency. By iteratively adjusting the capacity allocation scheme, it ultimately finds the optimal capacity allocation scheme that satisfies the latency constraint.

[0076] It should be noted that capacity allocation is necessary in this embodiment of the invention because resources at different storage tiers are limited. If storage tiers are only divided without capacity optimization, issues may arise such as insufficient capacity at the high-frequency access data tier, leading to high-frequency data being evicted from the cache, or redundant capacity at the low-frequency access data tier, resulting in resource waste. Therefore, precise capacity allocation ensures the access performance of data at each tier while keeping the total storage cost within an optimal range, achieving a dual improvement in resource utilization efficiency and business experience.

[0077] In this embodiment of the invention, by calling the storage optimization model and combining it with the simulated annealing algorithm to efficiently solve the optimal capacity allocation scheme, dynamic adaptation and adjustment of the capacity of each storage level is achieved. This method breaks through the experience limitations of manual allocation and can accurately match the storage needs of data with different attributes with the supply of resources at each level, avoiding the problem of redundancy or insufficiency of capacity at some levels.

[0078] Furthermore, the storage optimization model is constrained by an access latency not exceeding a preset latency upper limit, and the optimization objective is to minimize the total storage cost; wherein the total storage cost is obtained by summing the single-level storage cost of each storage tier, and the single-level storage cost is the product of the unit storage cost and storage capacity corresponding to the current storage tier.

[0079] For example, the model sets an upper limit threshold for access latency, meaning the combined access latency across all storage tiers must meet the performance tolerance requirements of the business side. For instance, AccessDelay ≤ 100ms means the user's wait time for data access does not exceed 100ms, which is a fundamental prerequisite for ensuring a good business experience. The model's core objective is to minimize total storage cost. The calculation process satisfies the following formula: (3); in, It is the unit storage cost at the SSD storage tier; This is the storage capacity allocated to this level; the product of these two is the single-level storage cost of the SSD level. This refers to the unit storage cost of HDFS, the object storage hierarchy. This is the allocated storage capacity, and the product of the two is the single-level storage cost of the HDFS hierarchy. The storage cost is the sum of the single-level storage costs of each hierarchy. The model finds the allocation scheme that minimizes the total storage cost by adjusting the storage capacity of different hierarchies while satisfying latency constraints.

[0080] In this embodiment of the invention, the storage optimization model uses an upper limit on access latency as a constraint to ensure that the response speed of core services such as high-frequency access data meets the standards and guarantees service quality. At the same time, it aims to minimize the total storage cost, achieving efficient resource utilization and precise cost control. Through dual-objective collaborative optimization, it avoids both performance deficiencies caused by simply pursuing cost reduction and waste caused by excessive resource investment, achieving a dynamic balance between storage performance and cost.

[0081] S5. The data to be stored is stored in the target storage level.

[0082] Further, step S5 specifically includes: when the target storage level is located at the central node, sending the data to be stored to the target storage level corresponding to the central node; when the target storage level is located at the edge node, after sending the data to be stored to the central node, the central node synchronizes the data to be stored to the target storage level corresponding to the edge node.

[0083] For example, if the target storage level of the data to be stored belongs to the central node, the data is directly sent to the central node, and the storage management module of the central node writes the data to the corresponding target storage level. For example, for a company's historical meeting recordings (low-frequency access data), the target storage level is the central node's HDFS object storage layer. In this case, the data will be directly transmitted to the central node, and the central node will complete the HDFS layer write operation without additional synchronization processes. If the target storage level belongs to the edge node, the data must first be sent to the central node to ensure global data consistency and backup reliability. Then, the central node pushes the data to the corresponding edge node and writes it to its target storage level. For example, for popular emoticons in 5G new calls (high-frequency access data), the target storage level is the lightweight SSD storage layer of a certain regional edge node. The data is first uploaded to the central node for backup, and then the central node synchronizes it to the SSD cache layer of the edge node in that region, ultimately achieving low-latency access for end users. At the same time, the backup of the central node can also avoid data loss due to edge node failure.

[0084] In this embodiment of the invention, the central node directly stores the data, while the edge nodes synchronously forward the data through the central node. This design ensures both the central node's global control and consistency verification of the data, and the edge nodes' proximity storage meets the low-latency access requirements for frequently accessed data. Furthermore, relying on the central node's synchronous backup avoids the reliability risks associated with direct storage at the edge nodes, achieving a balance between flexibility, security, and business adaptability in data storage under a distributed architecture, and improving the overall storage and retrieval efficiency of the system.

[0085] Furthermore, after storing the data to be stored, or after synchronizing the data to be stored to the target storage level corresponding to the edge node, the central node sends a notification message to the media plane associated with the data to be stored, so that the media plane retrieves the data from the corresponding storage level.

[0086] For example, see Figure 4 , Figure 4 This is a schematic diagram illustrating the process of a central node notifying the media plane to pull data, provided in an embodiment of the present invention. After the data to be stored is uploaded to the system from a third-party platform, it enters the resource processing central node. Upon receiving the data, the central node synchronizes it to the corresponding distributed nodes, i.e., edge nodes, as shown in the diagram as distributed node east and distributed node north. The synchronization process includes three stages: resource synchronization, resource verification, and resource cleanup. Resource synchronization transmits the data to the edge nodes; resource verification verifies data integrity; and resource cleanup deletes redundant or invalid data to ensure node storage efficiency. Edge nodes in different regions synchronize the data required by the corresponding regional businesses. When the central node completes data storage or data synchronization to the edge nodes, it sends a notification message to the UMF (Unified Media Function) media plane. The UMF media plane is the business display / call entry point for connecting to end users. In the diagram, it is divided by region: UMF01-A corresponds to the media plane of province A, and UMF32-C corresponds to the media plane of province C. Each regional node corresponds to 8 local media planes. The notification message specifies which distributed node the data is stored on, guiding the media plane to pull the data from the corresponding node. After receiving the notification, each UMF media plane pulls data from the corresponding distributed node. Since the media plane is physically close to the corresponding distributed node, the latency of the retrieval process is extremely low, which can quickly support the business requests of end users.

[0087] In this embodiment of the invention, after the central node completes storage or synchronization, it promptly sends a notification to the associated media plane to clarify the data storage location and retrieval path. This mechanism allows the media plane to accurately locate the data without actively querying, significantly shortening the resource retrieval response time and ensuring rapid business access.

[0088] Further, see Figure 5 , Figure 5 This is a flowchart of data synchronization between nodes provided in an embodiment of the present invention. After step S5 is executed, the method further includes steps S6 to S8.

[0089] S6. Obtain the vector clock of each stored data in each storage level, and use the vector clock to determine the change information of the stored data.

[0090] For example, a vector clock refers to an array-type data structure in a distributed system used to record the order of events and identify data versions, satisfying: VClock=(NodeID,Timestamp,Version), where NodeID represents the node identifier, Timestamp represents the timestamp, and Version represents the version number; each element corresponds to the clock information of a node, which can accurately mark the version status of data on different nodes. Change information refers to the incremental updates to the stored data, such as the parts of data that have been added, modified, or deleted, and is the core basis for ensuring data consistency among distributed nodes.

[0091] Furthermore, the step of using the vector clock to determine the change information of the stored data includes: comparing the vector clocks of the same stored data on different nodes; if the vector clocks of at least two nodes do not match, it is determined that the stored data has undergone an incremental change; obtaining the incremental data of the stored data that has undergone an incremental change, and performing a hash operation on the incremental data to generate hash fingerprint data; and determining the change information based on the hash fingerprint data.

[0092] For example, for the same stored data, the vector clocks (VClock) at different nodes are compared. For instance, the VClock of a certain emoji data is (Node01, 1690000000, V1) at the central node and (Node02, 1690000100, V2) at the edge node. Since the Timestamp and Version of the two nodes do not match, it is determined that the data has undergone an incremental change. The incremental change portion of this data is extracted as incremental data, and a hash function such as MurmurHash3 is used to calculate a unique hash fingerprint H(X), where X is the incremental data. The hash fingerprints of the incremental data at different nodes are compared. If the fingerprints match, it indicates that the incremental change content is synchronized and complete; if the fingerprints differ, the specific change content is located through the byte-level differences in the fingerprints, ultimately clarifying the data change information, including the changed content, the changed node, and the changed time.

[0093] In this embodiment of the invention, by comparing the vector clocks of different nodes, incremental data changes can be quickly and accurately determined. Combined with hash operation to generate fingerprint data to locate the specific changes, redundant transmission and high time consumption of full data comparison are avoided. Only incremental data processing is focused on, reducing synchronization bandwidth usage and system overhead. At the same time, it ensures that no change information is missed, providing an accurate basis for subsequent data synchronization and improving data consistency and collaboration efficiency among distributed nodes.

[0094] S7. Construct a synchronization queue model based on the data attributes and change frequency of the stored data, and use the synchronization queue model to generate the data synchronization scheduling order.

[0095] For example, the synchronization queue model refers to a state-action value association model Q(s,a) based on reinforcement learning, used to quantify the priority of data synchronization. Scheduling order refers to the sequence of data synchronization between distributed nodes. The reason for generating a scheduling order is that the amount of data to be synchronized in a distributed system is large. Unordered synchronization can lead to frequently changing hot data waiting too long, impacting business experience, or low-value data consuming bandwidth, wasting resources. Scheduling order can avoid these problems and achieve efficient resource allocation.

[0096] Furthermore, the synchronization queue model is a correlation model between state and action value; wherein, the state includes the data attribute and the change frequency, the action value corresponds to different levels of synchronization priority, and the synchronization queue model aims to maximize the expected return of the correlation function between the state and the action value to generate the scheduling order of data synchronization.

[0097] For example, this process achieves intelligent scheduling of synchronization priorities through reinforcement learning algorithms, such as the DDPG (Deep Deterministic Policy Gradient) algorithm. First, define a state `s` and an action `a`. State `s` contains core data features, such as data type, data attributes, change frequency (e.g., high-frequency data changing 10 times per hour), and node distance (physical distance between synchronization nodes). Action `a` corresponds to synchronization priority, divided into high, medium, and low levels. Then, construct the state-action value function `Q(s,a)`, where `Q(s,a)` is the expected reward for taking action `a` in state `s`. The higher the reward, the more reasonable the synchronization priority. For example, in a state `s` with high-frequency data access and high-frequency changes, taking a high-priority action `a` yields a higher reward; conversely, in a state `s` with low-frequency data access and low-frequency changes, taking a low-priority action `a` yields a higher reward. The DDPG algorithm is used to optimize the strategy. The model is trained using the DDPG algorithm with the goal of maximizing the expected return of Q(s,a). The model learns from historical synchronization data which priority to choose under different states to maximize overall synchronization efficiency, such as reducing synchronization latency for high-frequency access data and improving bandwidth utilization. Ultimately, it outputs the optimal synchronization priority strategy. Based on the priority output by the model, the data to be synchronized is sorted. For example, high-frequency access data of high-frequency changes in 5G new calls (state s corresponds to high reward) is assigned high priority and placed at the head of the synchronization queue, while archived low-frequency access data (state s corresponds to low reward) is assigned low priority and placed at the end of the queue.

[0098] In this embodiment of the invention, an association model is constructed using data attributes and change frequency as states and synchronization priority as action value. By maximizing expected returns, a scheduling order is generated, achieving intelligent allocation of synchronization resources. This model prioritizes high-frequency access data and high-frequency change data for synchronization resources, ensuring the consistency and low latency of core business data. At the same time, it avoids low-value data from consuming too much bandwidth, improving synchronization efficiency and resource utilization, and adapting to the dynamic synchronization needs of multiple types of data in distributed systems.

[0099] S8. Based on the scheduling order, synchronize the stored data.

[0100] For example, the data to be synchronized is arranged in scheduling order. The system prioritizes allocating bandwidth and computing resources to high-priority queues. After the high-priority data is synchronized, medium- and low-priority data are processed in sequence. During the synchronization process, the system feeds back information such as whether the synchronization was successful and resource usage to the reinforcement learning model, continuously optimizing the subsequent scheduling order strategy.

[0101] Furthermore, the data synchronization of the stored data includes: using the node containing the stored data that has not undergone incremental changes as the synchronization node; and sending the change information to the synchronization node.

[0102] For example, some frequently accessed data is initially stored in a central node and edge node A. Subsequently, this data in edge node A undergoes incremental changes, such as the addition of a new frame. The system recognizes that the data in the central node has not undergone incremental changes, so it uses the central node as a synchronization node. At this time, it only sends the incremental change information of the data in edge node A (such as the byte data of the new frame + hash fingerprint) to the central node, instead of transmitting the complete emoji file. After receiving the change information, the central node merges the incremental content based on the original data, completes the data update, and ultimately achieves data consistency between the central node and edge node A.

[0103] In this embodiment of the invention, nodes that have not undergone incremental changes are used as synchronization nodes, and change information is only sent to these nodes to avoid bandwidth waste and efficiency loss caused by full node synchronization.

[0104] In this embodiment of the invention, through steps S6-S8, the vector clock accurately captures data change information, constructs a synchronization queue model based on data attributes and change frequency, generates a reasonable scheduling order, and executes synchronization. This mechanism not only ensures real-time perception and consistency of data changes, but also prioritizes the synchronization of high-value data through priority scheduling, avoiding resource waste. Simultaneously, it improves the efficiency of data collaboration between distributed nodes, reduces synchronization latency and redundant transmission, enhances system data reliability and business adaptability, and adapts to the dynamic collaboration requirements of distributed storage.

[0105] See Figure 6 , Figure 6 This is a structural block diagram of a data storage device 100 provided in an embodiment of the present invention. The data storage device 100 includes: The data acquisition module 11 is used to acquire the data characteristics and data attributes of the data to be stored when it receives the data to be stored. The node information acquisition module 12 is used to acquire node information of all nodes in the resource center; The candidate node filtering module 13 is used to filter candidate nodes from all nodes based on the node information; The target node filtering module 14 is used to filter out target nodes that match the data to be stored based on the data features and the node information of the candidate nodes. Storage level determination module 15 is used to determine the target storage level corresponding to the target node based on the data attributes; The data storage module 16 stores the data to be stored to the target storage level.

[0106] Furthermore, the data storage device 100 also includes: The capacity adjustment module is used to call the storage optimization model; solve the storage optimization model using the simulated annealing algorithm to obtain the capacity allocation scheme for the storage level; and adjust the capacity of the storage level based on the capacity allocation scheme.

[0107] Furthermore, the data storage device 100 also includes: The change information determination module is used to obtain the vector clock of each stored data in each storage level, and use the vector clock to determine the change information of the stored data; The scheduling order determination module is used to construct a synchronization queue model based on the data attributes and change frequency of the stored data, and to generate a data synchronization scheduling order using the synchronization queue model. The data synchronization module is used to synchronize the stored data based on the scheduling order.

[0108] Furthermore, the data storage device 100 also includes: The data review module is used to review the content of the data to be stored.

[0109] It is worth noting that the working process of each module in the data storage device 100 described in the embodiments of the present invention can refer to the working process of the data storage method described in the above embodiments, and will not be repeated here.

[0110] See Figure 7 , Figure 7This is a structural block diagram of a data storage device 200 provided in an embodiment of the present invention. The data storage device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described embodiments of the various data storage methods.

[0111] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the data storage device 200.

[0112] The data storage device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the data storage device 200 and does not constitute a limitation on the data storage device 200. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the data storage device 200 may also include input / output devices, network access devices, buses, etc.

[0113] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the data storage device 200, connecting all parts of the data storage device 200 via various interfaces and lines.

[0114] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the data storage device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0115] If the modules / units integrated in the data storage device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0116] Furthermore, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the data storage method as described in any of the above embodiments.

[0117] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data storage method, characterized in that, include: Upon receiving data to be stored, the data characteristics and data attributes of the data to be stored are obtained; Obtain node information for all nodes in the resource center, and filter candidate nodes from all nodes based on the node information; Based on the data features and the node information of the candidate nodes, target nodes that match the data to be stored are selected. Determine the target storage level corresponding to the target node based on the data attributes; The data to be stored is stored in the target storage level.

2. The data storage method as described in claim 1, characterized in that, The process of obtaining the data features and data attributes of the data to be stored includes: The data to be stored is subjected to multimodal feature extraction using a data extraction model to obtain data features; The data attributes of the data to be stored are determined based on the data characteristics.

3. The data storage method as described in claim 2, characterized in that, The data characteristics include at least one of data size, access frequency, and data type.

4. The data storage method as described in claim 2, characterized in that, The data attributes include high-frequency access data, medium-frequency access data, and low-frequency access data; wherein, the high-frequency access data is data whose access frequency is in a first interval, the medium-frequency access data is data whose access frequency is in a second interval, and the low-frequency access data is data whose access frequency is in a third interval, the lower limit of the access frequency in the first interval is higher than the upper limit of the access frequency in the second interval, and the lower limit of the access frequency in the second interval is higher than the upper limit of the access frequency in the third interval.

5. The data storage method as described in claim 2, characterized in that, The data extraction model is a hybrid data extraction model composed of a Transformer model and a first neural network model; wherein, the Transformer model is used to extract data features of text data, and the first neural network model is used to extract data features of at least one type of data, including image data, video data, and audio data.

6. The data storage method as described in claim 1, characterized in that, The node information includes at least one of the following: node operating status, resource supply data, and deployment information.

7. The data storage method as described in claim 6, characterized in that, The step of filtering candidate nodes from all nodes based on the node information includes: Based on the network topology relationships and node information among all nodes, construct a node state graph; The node state graph is input into the second neural network model, and the scheduling priority matrix output by the second neural network model is obtained; All nodes are sorted based on the scheduling priority matrix, and the top n nodes in the priority sort are selected as candidate nodes, where n is an integer and n≥2.

8. The data storage method as described in claim 6 or 7, characterized in that, The step of filtering out target nodes that match the data to be stored based on the data features and the node information of the candidate nodes includes: Calculate the matching degree between the data features and the node information of each candidate node; The candidate node with the highest matching degree is selected as the target node for the data to be stored.

9. The data storage method as described in claim 4, characterized in that, Determining the target storage level corresponding to the target node based on the data attributes includes: If the data attributes are high-frequency access data and medium-frequency access data, the SSD storage layer is determined as the target storage layer corresponding to the target node; If the data attribute is low-frequency access data, the object storage layer is determined as the target storage layer corresponding to the target node.

10. The data storage method as described in claim 9, characterized in that, After determining the target storage level corresponding to the target node based on the data attributes, the method further includes: Invoke the storage optimization model; The storage optimization model is solved by the simulated annealing algorithm to obtain the capacity allocation scheme of the storage hierarchy; The capacity of the storage tier is adjusted based on the capacity allocation scheme.

11. The data storage method as described in claim 10, characterized in that, The storage optimization model is constrained by an access latency not exceeding a preset latency limit, and the optimization objective is to minimize the total storage cost.

12. The data storage method as described in claim 11, characterized in that, The total storage cost is obtained by summing the single-level storage cost of each storage tier, wherein the single-level storage cost is the product of the unit storage cost and the storage capacity corresponding to the current storage tier.

13. The data storage method as described in claim 1, characterized in that, After storing the data to be stored in the target storage level, the method further includes: Obtain the vector clock of each stored data in each storage level, and use the vector clock to determine the change information of the stored data; A synchronization queue model is constructed based on the data attributes and change frequency of the stored data, and the synchronization queue model is used to generate the scheduling order for data synchronization. Based on the scheduling order, the stored data is synchronized.

14. The data storage method as described in claim 13, characterized in that, The step of using the vector clock to determine the change information of the stored data includes: Compare the vector clocks of the same stored data on different nodes; If at least two nodes have mismatched vector clocks, it is determined that the stored data has undergone an incremental change; Obtain incremental data of stored data that has undergone incremental changes, and perform a hash operation on the incremental data to generate hash fingerprint data; The change information is determined based on the hash fingerprint data.

15. The data storage method as described in claim 13, characterized in that, The synchronization queue model is a correlation model between state and action value; wherein, the state includes the data attribute and the change frequency, the action value corresponds to different levels of synchronization priority, and the synchronization queue model aims to maximize the expected return of the correlation function between the state and the action value to generate the scheduling order of data synchronization.

16. The data storage method as described in claim 14, characterized in that, The process of synchronizing the stored data includes: The node containing the stored data that has not undergone incremental changes is used as the synchronization node; The change information is sent to the synchronization node.

17. The data storage method as described in claim 1, characterized in that, The resource center includes a central node and at least one edge node associated with the central node.

18. The data storage method as described in claim 17, characterized in that, The step of storing the data to be stored to the target storage level includes: When the target storage level is located at the central node, the data to be stored is sent to the target storage level corresponding to the central node; When the target storage level is located at the edge node, the data to be stored is sent to the central node, and then the central node synchronizes the data to be stored to the target storage level corresponding to the edge node.

19. The data storage method as described in claim 18, characterized in that, After storing the data to be stored, or after synchronizing the data to be stored to the target storage level corresponding to the edge node, the central node sends a notification message to the media plane associated with the data to be stored, so that the media plane can retrieve the data from the corresponding storage level.

20. The data storage method as described in claim 1, characterized in that, When receiving the data to be stored, the method further includes: The data to be stored is subject to content review.

21. A data storage device, characterized in that, include: The data acquisition module is used to acquire the data characteristics and data attributes of the data to be stored when it receives the data to be stored. The node information acquisition module is used to acquire node information of all nodes in the resource center; The candidate node filtering module is used to filter candidate nodes from all nodes based on the node information. The target node filtering module is used to filter out target nodes that match the data to be stored based on the data features and the node information of the candidate nodes. A storage level determination module is used to determine the target storage level corresponding to the target node based on the data attributes. The data storage module stores the data to be stored in the target storage level.

22. A data storage device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data storage method as described in any one of claims 1 to 20.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data storage method as described in any one of claims 1 to 20.

24. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the data storage method as described in any one of claims 1 to 20.

Images