A multi-modal distributed storage RDMA primitive optimization method, system and related device

Abstract

The application is suitable for the technical field of RDMA primitive optimization, and provides a multi-modal distributed storage RDMA primitive optimization method and system and related equipment, which comprises the following steps: receiving an access request through a multi-modal storage interface; analyzing data modal tags, QoS indicators and environmental parameters in the access request; matching an initial RDMA primitive combination corresponding to the data modal tags and the QoS indicators based on a pre-constructed mapping relationship rule library; dynamically optimizing the initial primitive combination through a machine learning model in combination with real-time collected network state data and environmental parameters to generate a target primitive combination; and issuing the target primitive combination to heterogeneous hardware resources to execute RDMA operations in a pipeline order, thereby effectively improving the performance and reliability of RDMA operations in a distributed storage system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of RDMA primitive optimization, and in particular relates to a multi-modal distributed storage RDMA primitive optimization method and system and related equipment. BACKGROUND

[0002] With the increasing demand for low latency and high throughput of distributed storage systems, remote direct memory access (RDMA) technology has gradually become a key means to optimize data access performance. Currently, RDMA primitives are usually statically classified based on a single dimension for data access requests, such as assigning small messages to SEND / RECV primitives, and large blocks of data to WRITE / READ primitives, and matching predefined primitive combinations based on fixed rules. However, when dealing with some streaming data, the burst high throughput demand may exceed the processing capacity of the fixed primitive combination. In addition, the existing primitive combination is difficult to cope with real-time network state and hardware resource changes. For example, when there is a burst network congestion, fixed use of the high-throughput READ primitive may cause a surge in packet retransmission, and the primitive cannot be dynamically switched to an atomic operation primitive that supports congestion control. That is, in the streaming data scenario, it is difficult to balance high throughput and low latency targets, and the utilization rate of heterogeneous hardware resources is low.

[0003] In view of this, there is a need for a multi-modal distributed storage RDMA primitive optimization method, system and related equipment. SUMMARY

[0004] The embodiments of the present application provide a multi-modal distributed storage RDMA primitive optimization method, system and related equipment, which are used to solve the problem of low utilization rate of heterogeneous hardware resources and difficulty in balancing high throughput and low latency targets in the streaming data scenario.

[0005] The first aspect of the embodiments of the present application provides a multi-modal distributed storage RDMA primitive optimization method, comprising:

[0006] receiving an access request through a multi-modal storage interface;

[0007] parsing data modality tags, QoS indicators and environment parameters in the access request, the data modality tags including structured data, unstructured data and streaming data, the QoS indicators including delay requirements, throughput requirements and consistency levels, and the environment parameters including network delay, packet loss rate, network card queue depth and hardware load;

[0008] matching an initial RDMA primitive combination corresponding to the data modality tags and QoS indicators based on a pre-constructed mapping relationship rule library;

[0009] Combine real-time network state data and the environment parameters, and dynamically optimize the initial RDMA primitive combination through a machine learning model to generate a target RDMA primitive combination;

[0010] Distribute the target RDMA primitive combination to heterogeneous hardware resources, and execute RDMA operations in a pipeline order, which is scheduled based on operation dependency and hardware resource parallelism.

[0011] Further, the data modality label, QoS indicator and environment parameter in the access request are analyzed, the data modality label includes structured data, unstructured data and stream data, the QoS indicator includes delay requirement, throughput demand and consistency level, and the environment parameter includes network delay, packet loss rate, network card queue depth and hardware load, including:

[0012] According to the data type, the transaction type and associated table of structured data, the metadata and BLOB fingerprint data of unstructured data, and the stream ID and timing index of stream data are analyzed to generate a modality identifier;

[0013] The delay threshold, throughput demand and consistency level are extracted from the access request header field, the structured data is mapped to the transaction isolation level, and the throughput of the stream data is dynamically calculated and the default consistency is set;

[0014] By measuring network delay, counting packet loss rate, monitoring network card queue depth and collecting hardware load, historical data and real-time data are fused;

[0015] The multi-dimensional parameters after analysis and fusion are packaged into a unified context object.

[0016] Further, the initial RDMA primitive combination corresponding to the data modality label and QoS indicator is matched based on a pre-constructed mapping relationship rule library, including:

[0017] The multi-dimensional features in the historical performance data are preprocessed, including encoding the data modality label into an enumeration value, normalizing the QoS indicator, and discretizing the environment parameter into low, medium and high level intervals;

[0018] Based on information gain, a split feature is selected from the multi-dimensional features;

[0019] A decision tree is recursively split until the number of node samples is less than a threshold, and each leaf node in the decision tree stores the RDMA primitive combination strategy with the highest comprehensive score, and the comprehensive score is the weighted sum of delay, throughput and error rate.

[0020] Further, the split feature is selected from the multi-dimensional features based on information gain, including:

[0021]

[0022] wherein: S is a dataset, A is a feature to be evaluated, S v is a subset of the feature A with value v, C is the number of categories of the RDMA primitive combination category, p i is the proportion of samples of the i-th category of RDMA primitive combination in the dataset.

[0023] Further, the matching of the initial RDMA primitive combination corresponding to the data modality label and the QoS indicator based on the pre-constructed mapping relationship rule library further comprises:

[0024] When detecting a change in hardware resources or adding a new RDMA primitive type, collect new historical performance data and retrain the decision tree model;

[0025] Perform conflict detection on the original leaf node strategy. If the difference in QoS indicators between the new and old strategies exceeds a threshold, replace it with a new strategy with higher confidence;

[0026] Serialize the updated decision tree model into a new mapping relationship rule library, and replace the old rule library in runtime through a hot loading mechanism.

[0027] Further, the combination of real-time collected network state data and the environment parameters, and the dynamic optimization of the initial RDMA primitive combination through the machine learning model to generate the target RDMA primitive combination, comprises:

[0028] Real-time collection of network bandwidth, packet loss rate and hardware load data, construction of a normalized feature vector containing data modality, QoS indicators and initial primitive encoding;

[0029] Input the normalized feature vector into a pre-trained deep Q network model, and select the optimal adjustment action by maximizing the objective function;

[0030] Generate a target RDMA primitive combination according to the model output action, the target RDMA primitive combination includes primitive replacement, parameter tuning or mixed mode allocation, and submit the target RDMA primitive combination to the production environment after verification in the sandbox environment.

[0031] Further, the generation of the target RDMA primitive combination according to the model output action, the target RDMA primitive combination including primitive replacement, parameter tuning or mixed mode allocation, and the submission of the target RDMA primitive combination to the production environment after verification in the sandbox environment, comprises:

[0032] compiling the compute-intensive operations in the target RDMA primitive combination into CUDA kernels, and assigning to the stream multiprocessors of the GPU for parallel execution;

[0033] converting the data transfer primitives into DMA descriptor chains of the DPU through the NVIDIA DOCA SDK, and registering to the network card queue;

[0034] constructing a DAG scheduling graph based on operation dependency, and splitting tasks of independent nodes according to hardware parallelism.

[0035] Further, the issuing of the target RDMA primitive combination to heterogeneous hardware resources, and the execution of RDMA operations in pipeline order, the pipeline order being scheduled based on operation dependency and hardware resource parallelism, comprises:

[0036] constructing a directed acyclic graph based on the operation semantics of the target RDMA primitive combination, the nodes representing RDMA operations, and the edges representing data dependency between operations;

[0037] calculating the total delay of each path in the directed acyclic graph using a critical path first scheduling algorithm, and preferentially assigning operations on the longest path to low-delay hardware resources;

[0038] dynamically splitting tasks of independent nodes according to hardware resource parallelism;

[0039] assigning the split tasks to the multi-queues of GPUs, DPUs and smart network cards, and synchronously inserting hardware barrier instructions.

[0040] The second aspect of the embodiments of the present application provides a multi-modal distributed storage RDMA primitive optimization system, comprising:

[0041] an access request receiving unit configured to receive an access request through a multi-modal storage interface;

[0042] a data parameter parsing unit configured to parse data modal label, QoS indicator and environment parameter in the access request, the data modal label comprising structured data, unstructured data and stream data, the QoS indicator comprising delay requirement, throughput demand and consistency level, and the environment parameter comprising network delay, packet loss rate, network card queue depth and hardware load;

[0043] an initial RDMA primitive combination matching unit configured to match an initial RDMA primitive combination corresponding to the data modal label and QoS indicator based on a pre-constructed mapping relationship rule library;

[0044] a target RDMA primitive combination generation unit configured to combine real-time collected network state data and the environment parameters, and dynamically optimize the initial RDMA primitive combination through a machine learning model to generate a target RDMA primitive combination;

[0045] an RDMA operation execution unit configured to issue the target RDMA primitive combination to heterogeneous hardware resources, and execute RDMA operations in a pipeline order, the pipeline order being scheduled based on operation dependency and hardware resource parallelism.

[0046] A third aspect of the embodiments of the present application provides a computer device, including a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, and the processor executes the computer readable instructions to implement the steps of the RDMA primitive optimization method of the multi-modal distributed storage as described in any one of the above aspects.

[0047] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0048] Based on multi-dimensional feature analysis of data modalities, QoS and environment parameters, the mapping relationship between multi-modal data and primitives is determined based on a pre-constructed mapping relationship rule library, and the optimal primitive combination of structured data strong consistency, high throughput of stream data and other scenarios is matched from multiple dimensions. Then, through a reinforcement learning model, network status and hardware load are fused in real time, and the primitive combination and parameters are dynamically adjusted, which can automatically switch to a degraded strategy in the event of sudden congestion or hardware failure. Through multi-modal data dynamic classification, rule library and machine learning collaborative optimization, and heterogeneous hardware pipeline scheduling, the performance and reliability of RDMA operations in the distributed storage system are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 Figure 1 is a flowchart of an embodiment of the RDMA primitive optimization method of the multi-modal distributed storage in the present application. DETAILED DESCRIPTION

[0050] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, and above-described drawings, if any, are used to distinguish between similar objects and are not necessarily used to describe a particular sequential or chronological order. It is to be understood that the use of the terms so-termed, where appropriate, can be interchanged with each other to the extent that the embodiments of the application described herein can be implemented in other than the order depicted or described herein. Furthermore, the terms "comprise" and "correspond to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or apparatuses.

[0051] Embodiment one

[0052] The method implemented in this embodiment can be implemented in a system, which can be implemented in a server or a terminal, and the specific implementation is not limited. From the perspective of system implementation, the RDMA primitive optimization method of multi-modal distributed storage in this application will be introduced. Please refer to Figure 1 The method provided by the embodiment of the present application comprises the following steps:

[0053] S11. Receive an access request through a multi-modal storage interface;

[0054] In this embodiment, the access request includes structured data, unstructured data and stream data, and the multi-modal storage interface receives the access request, which is based on multi-protocol adaptation, dynamic dispatching and metadata enhancement. Specifically, the interface supports HTTP, RDMA-native Verbs and other protocols based on a unified communication framework, automatically parses structured data, unstructured data and stream data, and ensures the integrity and security of the request through CRC check and RBAC authentication. Then, according to the data type, it is dispatched to the corresponding processing queue, which includes a high consistency transaction queue, a large block data cache area and a low delay ring buffer; at the same time, data modal labels, timestamps and other metadata are injected into the request header. Here, by compatible multi-modal data characteristics, security access and pre-classification are adapted to the subsequent processing flow, and finally the high throughput and low delay request receiving, accurate resource pre-allocation and end-to-end tracing capability are realized, which provides a data basis for the subsequent steps.

[0055] S12. Analyze the data modal label, QoS indicator and environmental parameter in the access request, the data modal label includes structured data, unstructured data and stream data, the QoS indicator includes delay requirement, throughput demand and consistency level, and the environmental parameter includes network delay, packet loss rate, network card queue depth and hardware load;

[0056] Step S12 comprises the following:

[0057] S121. According to the data type, the transaction type and the association table of the structured data are parsed respectively, the metadata and the BLOB fingerprint data of the unstructured data, the stream ID and the time sequence index of the stream data are generated, and the modal identifier is generated. Here, the BLOB fingerprint data is a unique identifier generated by a hash algorithm for a binary large object (BLOB), which is used to represent the integrity and uniqueness of the BLOB content. Its essence is a fixed-length hash value, which can uniquely map the original data content. Even if the data changes slightly, the fingerprint will be significantly different;

[0058] According to the data type, the key features are extracted, and a unified modal identifier is generated, which provides a classification basis for subsequent RDMA primitive matching.

[0059] Structured data parsing includes: parsing the operation_type field from the Protobuf / JSON payload, such as SELECT, UPDATE, and combining the SQL syntax tree analysis to determine the transaction type; extracting the table name and foreign key relationship in the SQL statement to generate a Schema hash value for identifying data association; and finally outputting the identifier.

[0060] Unstructured data parsing includes: reading Content-Type, Content-Length and custom labels from the object storage request header; calculating SHA-256 hash for binary large object (BLOB) to generate a unique fingerprint; and finally outputting the identifier.

[0061] Stream data parsing includes: extracting Stream ID (such as stream_123), sequence number and timestamp from the data packet header, generating time sequence index according to time window (such as 1 second); reading TCP window size or Kafka partition ID to determine stream priority; and finally outputting the identifier.

[0062] S122. Extract the delay threshold, throughput demand and consistency level from the access request header field, map the transaction isolation level for structured data, dynamically calculate the throughput for stream data and set the default consistency;

[0063] This step combines explicit QoS requirements with implicit rules, converts them into quantifiable parameters, and guides the primitive combination optimization. Specifically, the delay requirement analysis includes: reading the maximum allowed delay from the request header X-QoS-Latency field; if not explicitly specified, set the default value according to the data mode; structured data is strong consistency transaction default and weak consistency query default, and stream data is real-time processing default and batch processing default. Calculate the throughput requirement according to the operation type of unstructured data, and estimate the throughput requirement according to the sampling rate and single sample size of stream data. The consistency level is mapped according to the transaction isolation level of structured data, including weak consistency and strong consistency; stream data is weak consistency by default, and is upgraded to strong consistency if the request header contains X-Consistency-Level.

[0064] S123. By measuring network delay, statistical packet loss rate, monitoring network card queue depth and collecting hardware load, historical data and real-time data are fused;

[0065] This step improves the robustness of the optimization strategy by collecting network and hardware status in real time and combining historical data to predict short-term environmental changes. Fusion includes aligning historical sliding window data with real-time acquisition values by timestamp and removing transient noise.

[0066] S124. Encapsulate the multi-dimensional parameters after analysis and fusion into a unified context object.

[0067] Finally, the analysis results are integrated into a unified data structure, and the context object is written into shared memory or DMA mapping area to avoid multiple memory copies. Cache the context object of high-frequency requests for 5 seconds to reduce repeated analysis overhead.

[0068] S13. Based on the pre-constructed mapping relationship rule library, match the initial RDMA primitive combination corresponding to the data mode label and QoS index;

[0069] Step S13 includes the following:

[0070] S131. Preprocess the multi-dimensional features in the historical performance data, including encoding the data mode label into an enumeration value, normalizing the QoS index, and discretizing the environment parameters into low, medium and high level intervals;

[0071] Encoding is structured data→enumeration value 0, unstructured data→1, stream data→2; normalization processing is linear mapping and logarithmic scaling; converting the original heterogeneous data into a standardized format suitable for decision tree training, which can eliminate dimension differences and improve model convergence efficiency.

[0072] S132. Select the split feature from the multi-dimensional features based on information gain;

[0073]

[0074] Wherein: S is a data set, A is a feature to be evaluated, S v is a subset of the feature A with a value of v, C is the number of categories of the RDMA primitive combination category, p i is the sample proportion of the i-th category of RDMA primitive combination in the data set.

[0075] The calculated features are sorted in descending order of gain, for example, data mode is greater than delay, delay is greater than throughput.

[0076] S133. Recursively split to generate a decision tree until the number of samples in the node is less than a threshold, and each leaf node in the decision tree stores the RDMA primitive combination strategy with the highest comprehensive score, and the comprehensive score is the weighted sum of delay, throughput and error rate.

[0077] The feature with the highest gain and the split point are selected as the split condition, and the stop condition is set to the number of node samples being less than 5% of the total samples or the tree depth being greater than or equal to 5 layers. The score of the leaf node is calculated, and the primitive combination with the highest score is selected as the initial strategy.

[0078] Specifically, the dynamic updating method of the rule base includes the following:

[0079] 1. When a hardware resource change or a new RDMA primitive type is detected, collect new historical performance data and retrain the decision tree model;

[0080] 2. Conflict detection is performed on the original leaf node strategy, and if the difference in QoS indicators between the new and old strategies exceeds a threshold, the new strategy with higher confidence is replaced;

[0081] 3. The updated decision tree model is serialized into a new mapping relationship rule base, and the old rule base in runtime is replaced through a hot loading mechanism.

[0082] When a new hardware or a new primitive is detected, record the performance data in the new scenario, including features and labels, and incrementally train or fully retrain on the original decision tree. The QoS difference threshold of the new and old strategies is used for conflict judgment, and only when the confidence of the new strategy is ≥ 90% is the old strategy replaced. The new rule base is loaded into the memory standby area, the pointer is atomically switched to point to the new base, and the old base resources are asynchronously released.

[0083] S14. Combine the real-time collected network state data and environmental parameters, and dynamically optimize the initial RDMA primitive combination through a machine learning model to generate a target RDMA primitive combination;

[0084] Step S14 includes the following:

[0085] S141. Real-time collection of network bandwidth, packet loss rate and hardware load data, construction of a normalized feature vector containing data modalities, QoS indicators and initial primitive encodings;

[0086] S142. Input the normalized feature vector into the pre-trained deep Q network model, and select the optimal adjustment action by maximizing the objective function;

[0087] The normalized data modalities, QoS indicators, and initial primitive encodings are spliced into a unified feature vector, which is input into the pre-trained DQN model. The model architecture includes an input layer, a hidden layer, and an output layer. The objective function is: Q(s,a) = α·T + β·(1 / L)―γ·E, where T is the throughput gain, L is the delay penalty, E is the error rate, and α, β and γ are the corresponding weight coefficients. Finally, the ε-greedy strategy is used to select the action with the highest Q value with a probability of 90%, and to randomly explore new strategies with a probability of 10%.

[0088] S143. Generate target RDMA primitive combinations according to the model output actions, which include primitive replacement, parameter tuning or mixed mode allocation, and submit the target RDMA primitive combinations to the production environment after verification in the sandbox environment.

[0089] The generated target RDMA primitive combination corresponds to the strategy in the independent RDMA channel, which is simulated by the target combination. The verified target combination is written to the RDMA queue, and the client is notified to switch to the new strategy. Specifically, it includes the following:

[0090] 1. Compile the compute-intensive operations in the target RDMA primitive combination into CUDA kernels, and assign them to the stream multiprocessors of the GPU for parallel execution;

[0091] 2. Convert the data transfer primitive to a DMA descriptor chain of DPU through NVIDIA DOCA SDK, and register it to the network card queue;

[0092] 3. Based on the operation dependency, build a DAG scheduling graph, and split the tasks according to the hardware parallelism for the independent nodes.

[0093] Through the hot switching mechanism, the target combination is seamlessly issued, and the compute-intensive operations are compiled into CUDA kernels, DPU descriptor chains and DAG scheduling tasks, realizing end-to-end low-latency high-throughput execution, and real-time monitoring of performance feedback to model iteration optimization.

[0094] S15. Issue the target RDMA primitive combination to heterogeneous hardware resources, and execute the RDMA operations in pipeline order. The pipeline order is scheduled based on operation dependency and hardware resource parallelism.

[0095] S151. Construct a directed acyclic graph based on the operation semantics of the target RDMA primitive combination, with nodes representing RDMA operations and edges representing data dependency relationships between operations;

[0096] S152. Calculate the total delay of each path in the directed acyclic graph using a critical path first scheduling algorithm, and preferentially allocate operations on the longest path to low-delay hardware resources;

[0097] S153. Dynamically split tasks of independent nodes according to the parallelism of hardware resources;

[0098] S154. Assign the split tasks to the multi-queues of GPU, DPU and smart NIC, and synchronously insert hardware barrier instructions.

[0099] The above steps are as follows: based on the analysis of the target RDMA primitive combination, label the maximum parallelism for each node. Preferentially schedule the operations on the longest path in the DAG to minimize the overall delay; allocate compute-intensive operations to GPU / DPU and high-throughput continuous transmission to smart NIC multi-queue; dynamically migrate tasks to low-load devices according to real-time hardware load. Generate operation execution order, for each operation node, select hardware from the resource pool that meets the support operation type and the current load is less than the maximum capacity threshold. Use a greedy algorithm to assign a time window to the operation to ensure maximum parallelization of independent operations. Convert the RDMA primitive to the underlying instruction of the target hardware, generate the DMA descriptor chain through the DOCA SDK, and compile it into a CUDA kernel; map the data buffer address directly to the hardware access space through GPUDirect RDMA to avoid CPU memory copy. Ensure result serialization through global memory lock, and if any operation fails, trigger rollback along the DAG in reverse.

[0100] The above embodiment dynamically classifies multi-modal data, optimizes rules library and machine learning in coordination, and schedules heterogeneous hardware pipeline. The selected primitive combination can cope with real-time network state and hardware resource changes, effectively improving the performance and reliability of RDMA operations in distributed storage systems.

[0101] Embodiment two

[0102] An embodiment of a multi-modal distributed storage RDMA primitive optimization system in the application includes the following steps:

[0103] An access request receiving unit is configured to receive access requests through a multi-modal storage interface;

[0104] A data parameter analysis unit is configured to analyze a data mode label, a QoS indicator and an environment parameter in the access request, the data mode label including structured data, unstructured data and stream data, the QoS indicator including a delay requirement, a throughput demand and a consistency level, and the environment parameter including a network delay, a packet loss rate, a network card queue depth and a hardware load.

[0105] An initial RDMA primitive combination matching unit is configured to match an initial RDMA primitive combination corresponding to the data mode label and the QoS indicator based on a pre-constructed mapping relationship rule library.

[0106] A target RDMA primitive combination generation unit is configured to dynamically optimize the initial RDMA primitive combination through a machine learning model in combination with real-time collected network state data and the environment parameter, and generate a target RDMA primitive combination.

[0107] An RDMA operation execution unit is configured to issue the target RDMA primitive combination to heterogeneous hardware resources, and execute RDMA operations in a pipeline order, the pipeline order being scheduled based on operation dependency and hardware resource parallelism.

[0108] Embodiment three

[0109] The application provides a computer device, including a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, and the processor executes the computer readable instructions to implement the steps of the above method.

[0110] Those skilled in the art can appreciate that the units of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both, and the constitution of the examples has been described in general in the above description in order to clearly illustrate the interchangeability of hardware and software. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0111] In the embodiments provided in the application, it should be understood that the division of units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored. In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0112] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0113] It can be understood that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application.

Claims

1. A method for multi-modal distributed storage RDMA primitive optimization, the method comprising: The method comprises the following steps: receiving an access request through a multi-modal storage interface; parsing the data modal label, QoS indicator and environment parameter in the access request, the data modal label including structured data, unstructured data and stream data, the QoS indicator including delay requirement, throughput demand and consistency level, and the environment parameter including network delay, packet loss rate, network card queue depth and hardware load; matching an initial RDMA primitive combination corresponding to the data modal label and QoS indicator based on a pre-constructed mapping relationship rule library, including: preprocessing multi-dimensional features in historical performance data, the preprocessing including encoding the data modal label into an enumeration value, normalizing the QoS indicator, and discretizing the environment parameter into low, medium and high level intervals; selecting a split feature from the multi-dimensional features based on information gain; recursively splitting to generate a decision tree until the number of node samples is less than a threshold, each leaf node in the decision tree storing an RDMA primitive combination strategy with the highest comprehensive score, the comprehensive score being a weighted sum of delay, throughput and error rate; when detecting hardware resource changes or adding new RDMA primitive types, collecting new historical performance data and retraining the decision tree model; performing conflict detection on the original leaf node strategy, and if the QoS indicator difference between the new and old strategies exceeds a threshold, replacing the old strategy with the new strategy with higher confidence; serializing the updated decision tree model into a new mapping relationship rule library and replacing the old rule library in runtime through a hot loading mechanism; combining real-time collected network state data and the environment parameter, dynamically optimizing the initial RDMA primitive combination through a machine learning model to generate a target RDMA primitive combination, including: collecting network bandwidth, packet loss rate and hardware load data in real time, and constructing a normalized feature vector containing data modal, QoS indicator and initial primitive code; inputting the normalized feature vector into a pre-trained deep Q network model to select an optimal adjustment action by maximizing an objective function; the model architecture comprises an input layer, a hidden layer and an output layer, and the objective function is: wherein, is a throughput gain, is a delay penalty, is an error rate, , and are corresponding weight coefficients, respectively. generating a target RDMA primitive combination according to the model output action, the target RDMA primitive combination including primitive replacement, parameter tuning or mixed mode allocation, and submitting the target RDMA primitive combination to the production environment after verification in a sandbox environment; downloading the target RDMA primitive combination to heterogeneous hardware resources, executing RDMA operations in pipeline order, the pipeline order being scheduled based on operation dependency and hardware resource parallelism.

2. The multi-modal distributed storage RDMA primitive optimization method of claim 1, wherein, The parsing of the data modal label, QoS indicator and environment parameter in the access request, the data modal label including structured data, unstructured data and stream data, the QoS indicator including delay requirement, throughput demand and consistency level, and the environment parameter including network delay, packet loss rate, network card queue depth and hardware load, includes: respectively parsing the transaction type and associated table of structured data, the metadata and BLOB fingerprint data of unstructured data, and the stream ID and timing index of stream data to generate a modal identifier; extracting the delay threshold, throughput demand and consistency level from the access request header field, mapping the structured data to the transaction isolation level, and dynamically calculating the throughput of stream data and setting the default consistency; By measuring network delay, packet loss rate, monitoring network card queue depth and collecting hardware load, historical data and real-time data are fused; The multi-dimensional parameters after analysis and fusion are packaged as a unified context object.

3. The multi-modal distributed storage RDMA primitive optimization method of claim 1, wherein, The split feature is selected from the multi-dimensional features based on information gain, including: wherein: is a dataset, is a feature to be evaluated, is a feature is a value is a subset of , is a number of categories of RDMA primitive combination categories, is a sample proportion of the RDMA primitive combination category.

4. The multi-modal distributed storage RDMA primitive optimization method of claim 1, wherein, The target RDMA primitive combination is generated according to the model output action, including primitive replacement, parameter optimization or mixed mode allocation, and the target RDMA primitive combination is submitted to the production environment after being verified in the sandbox environment, including: The computation-intensive operation in the target RDMA primitive combination is compiled into a CUDA kernel and executed in parallel on the stream multi-processor of the GPU; The data transfer primitive is converted into a DMA descriptor chain of the DPU and registered to the network card queue; A DAG scheduling graph is constructed based on operation dependency, and tasks of independent nodes are split according to hardware parallelism.

5. The multi-modal distributed storage RDMA primitive optimization method of claim 1, wherein, The target RDMA primitive combination is issued to heterogeneous hardware resources to execute RDMA operations in pipeline order, and the pipeline order is scheduled based on operation dependency and hardware resource parallelism, including: A directed acyclic graph is constructed based on the operation semantics of the target RDMA primitive combination, nodes represent RDMA operations, and edges represent data dependency between operations; A critical path first scheduling algorithm is used to calculate the total delay of each path in the directed acyclic graph, and operations on the longest path are preferentially allocated to low-latency hardware resources; Tasks of independent nodes are dynamically split according to hardware resource parallelism; The split tasks are allocated to the multi-queues of GPU, DPU and smart NIC, and hardware barrier instructions are inserted synchronously.

6. A multi-modal distributed storage RDMA primitive optimization system, comprising: The multi-modal distributed storage RDMA primitive optimization method of any one of claims 1-5 is used, including: An access request receiving unit is configured to receive an access request through a multi-modal storage interface; A data parameter analysis unit is configured to analyze data modal labels, QoS indicators and environment parameters in the access request, the data modal labels including structured data, unstructured data and stream data, the QoS indicators including delay requirements, throughput requirements and consistency levels, and the environment parameters including network delay, packet loss rate, network card queue depth and hardware load; An initial RDMA primitive combination matching unit is configured to match an initial RDMA primitive combination corresponding to the data modal labels and QoS indicators based on a pre-constructed mapping relationship rule library; A target RDMA primitive combination generation unit is configured to dynamically optimize the initial RDMA primitive combination through a machine learning model to generate a target RDMA primitive combination in combination with real-time collected network state data and the environment parameters; An RDMA operation execution unit is configured to issue the target RDMA primitive combination to heterogeneous hardware resources to execute RDMA operations in pipeline order, and the pipeline order is scheduled based on operation dependency and hardware resource parallelism. 7.A computer device, comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, wherein, The processor executes the computer readable instructions to implement the steps of the multi-modal distributed storage RDMA primitive optimization method of any one of claims 1-5.

Images