Storage and calculation integrated data scheduling system and method for high-concurrency scene

Abstract

The invention relates to the technical field of high-concurrency data processing, and discloses a storage and calculation integrated data scheduling system and method for a high-concurrency scene, and the system comprises a storage and calculation fusion architecture module, a distributed cache module, a dynamic fragmentation module, a parallel calculation module, a preloading module and a resource scheduling center. The method corresponds to the system. According to the method, by deeply fusing data storage and calculation and utilizing a distributed caching technology and a data preloading mechanism, I / O bottleneck is effectively reduced, dynamic data fragmentation and parallel calculation are supported, and the method is particularly suitable for large-scale data analysis scenes.

Description

Technical Field

[0001] The present application relates to the field of high-concurrency data processing technology, and specifically to a storage-computing integrated data scheduling system and method for high-concurrency scenarios. Background Art

[0002] Traditional architectures like the von Neumann architecture face serious challenges in high-concurrency scenarios. Frequent data transfers between memory and the processor lead to significant latency and energy consumption. As data volumes grow exponentially, memory speed increases far outstrip processor performance gains. Consequently, the processor spends significant time waiting for memory read and write operations to complete, creating a performance bottleneck.

[0003] For example, Chinese patent publication number CN119493833A discloses a data processing method, device, and electronic device for high-concurrency scenarios. This architecture employs a storage-computing separation, reducing database read pressure by optimizing data acquisition methods (local cache, in-memory database). However, this solution still suffers from the frequent data transfer between storage and computing units, failing to address the I / O bottleneck in large-scale parallel computing scenarios.

[0004] Another example is a Chinese patent application with publication number CN119066022A, which discloses a storage-computing integrated system, storage-computing functional unit, and device. This architecture embeds the storage-computing functional unit into the processor, connecting it via non-bus lines to reduce processor computing pressure. However, this solution focuses primarily on hardware-level optimization and lacks support for dynamic data sharding and parallel computing for high-concurrency scenarios.

[0005] Based on the above-mentioned storage-computing separation architecture and storage-computing integration architecture, it can be seen that the existing technology still has the following problems: First, the data transmission delay is high, especially when processing a large number of concurrent requests; second, the system scalability is limited, and it is difficult to cope with sudden traffic; third, there is a lack of intelligent preloading mechanism, and it is impossible to actively predict and load hot data; fourth, the sharding strategy is decoupled from the allocation of computing resources, and dynamic collaboration cannot be achieved. Summary of the Invention

[0006] The purpose of this application is to provide a storage-computing integrated data scheduling system and method for high-concurrency scenarios to solve the data transmission delay and I / O bottleneck problems in high-concurrency scenarios. By combining the storage-computing integrated architecture with dynamic data sharding technology, efficient data scheduling and parallel computing can be achieved, thereby improving the overall system performance and service quality.

[0007] To achieve the above objectives, this application discloses the following technical solutions: In a first aspect, the present application discloses a storage and computing integrated data scheduling system for high-concurrency scenarios, comprising: The storage and computing fusion architecture module is configured to: couple the data storage unit with the computing unit to enable the coordinated operation of the data storage function and the computing function; The distributed cache module is configured to cache data using a multi-level cache strategy, wherein the multi-level cache strategy includes: local cache, memory database and remote storage; The dynamic sharding module is configured to perform data sharding based on consistent hashing and dynamic weight calculation; The parallel computing module is configured to coordinate computing tasks with data sharding distribution so that data can be processed directly on the storage nodes; The preloading module is configured to: analyze historical request patterns and real-time traffic characteristics based on machine learning models, predict hot data, and load it into the cache layer in advance; The resource scheduling center is configured to perform resource discovery and data transmission through a combination of CXL and RELOAD protocols. The CXL protocol is used for memory-level low-latency interaction, and the RELOAD protocol is used for cross-node routing and resource discovery.

[0008] Preferably, the distributed cache module includes: The level adjustment unit is configured to: automatically adjust the cache level according to the data access frequency; The cache management unit is configured to optimize cache efficiency based on a setting of a cache expiration time.

[0009] Preferably, the preloading module is specifically configured as follows: Based on machine learning models, periodic predictions are made based on time series, analyzing periodic patterns in historical request patterns; Quantify the hotness of data based on the weight of access frequency; Burst traffic based on QPS thresholds triggers preloading at traffic peaks.

[0010] Preferably, the steps of implementing the data sharding include: Create a new shard task and obtain the recommended shard size, recommended concurrency, and unique ID of the corresponding upload task; Check the latest upload speed data, if there is the latest upload speed data of the upload task, calculate the shard size and the number of concurrent connections as the allocation scheme, if not, use the corresponding recommended shard size and the recommended number of concurrent connections as the allocation scheme; Execute uploads according to the allocation plan and record the uploaded shard size, number of concurrent uploads, and upload time; Based on the fragment upload results, confirm whether all fragments have been uploaded successfully. If successful, merge the fragments to complete the task.

[0011] Preferably, the step of implementing the data sharding further includes: Based on the multi-part upload results, confirm whether all parts have been uploaded successfully. If not, perform the following steps: A. Identify the type of failed shards, including network outage, node overload, or storage anomaly, and record a failure log, including the shard ID, failure time, node status, and error code. B. Analyze the real-time load status of the failed node based on dynamic weight calculation to determine whether the upload failure is caused by resource overload; C. Automatically retry the failed shards a preset number of times, dynamically adjusting the shard size or concurrency based on the computing task type and load of the current node GPU during the retry. D. If multiple retries fail, recalculate the dynamic weight, distribute the shards to other nodes, and update the shard upload strategy; E. If the failed node remains unavailable, the RELOAD protocol is used to query available node resources and migrate the unfinished shards to the new node for continued upload. During the migration process, the cache consistency mechanism of the CXL protocol is used to synchronize the data status between the old and new nodes. F. If the failed shard is hot data, the preloading module triggers the emergency preloading strategy, loads the relevant data into the cache layer of the standby node in advance, and dynamically adjusts the preloading range in combination with the QPS threshold.

[0012] Preferably, the remote storage is a NOF+ network.

[0013] Preferably, the resource scheduling center is further configured to: acquire data using NOF+ technology and synchronize the results to other nodes.

[0014] Preferably, the parallel computing module is further configured to: Under the AB dual-plane networking architecture with a scale of thousands of nodes, computing tasks are allocated through a dynamic load balancing algorithm, and the task granularity is adjusted according to the execution efficiency of GPU kernel functions; and when executing task decomposition, cross-node resource migration is performed, and the task allocation strategy is dynamically adjusted using the RELOAD protocol.

[0015] Preferably, during cross-node communication, the resource scheduling center utilizes a shared memory mechanism via the CXL protocol to achieve inter-node cache consistency; and employs a lightweight heartbeat packet via the RELOAD protocol to maintain node status.

[0016] In a second aspect, the present application discloses a storage-computing integrated data scheduling method for high-concurrency scenarios, which is applied to the storage-computing integrated data scheduling system for high-concurrency scenarios as described above. The method includes the following steps: System initialization: Establish a distributed storage and computing node network, where each node includes a storage unit and a computing unit; Data sharding: Dynamically distribute data to different nodes based on consistent hashing algorithm and dynamic weight calculation; Load monitoring: collects CPU utilization, memory utilization, memory bandwidth, core temperature, I / O latency, and memory usage of each node in real time; Predictive analysis: Analyze historical request patterns and real-time traffic characteristics based on machine learning models to predict hotspot data; Preload execution: Based on the prediction results of hot data, the hot data is loaded into the cache layer of the corresponding node in advance; Task allocation: Coordinate computing tasks with data sharding allocation, allowing data to be processed directly on storage nodes; Result merging: Collect the calculation results of each node, merge them into the final result and return it to the user.

[0017] Beneficial effects: The storage-computing integrated data scheduling system and method for high-concurrency scenarios of the present application reduces data transmission paths and lowers I / O delays through the storage-computing fusion architecture module; realizes localized calculation of data on storage nodes through the parallel computing module, reduces cross-node communication overhead, and improves overall computing efficiency; realizes real-time adjustment of shard size and concurrency number through the dynamic sharding module to avoid resource overload or idleness; realizes automatic adjustment of cache levels driven by hot data access frequency through the distributed cache module to improve hit rate; realizes early loading of potential hot data through the preloading module to improve response speed; realizes memory-level low-latency interaction and cross-node routing maintenance through the resource scheduling center to ensure cache consistency and dynamic topology updates, and meets real-time requirements in high-concurrency scenarios through the CXL+RELOAD protocol combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a structural block diagram of the storage and computing integrated data scheduling system for high-concurrency scenarios provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following is a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] In this document, the term "comprising" is intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0022] In a first aspect, this embodiment provides a Figure 1 The storage-computing integrated data scheduling system for high-concurrency scenarios shown in the figure deeply integrates data storage and computing, utilizes distributed caching technology and data preloading mechanisms, effectively reduces I / O bottlenecks, supports dynamic data sharding and parallel computing, and is particularly suitable for large-scale data analysis scenarios. The system includes: The storage and computing fusion architecture module is configured to: couple the data storage unit with the computing unit to enable the coordinated operation of the data storage function and the computing function, thereby reducing the data transmission path; The distributed cache module is configured to cache data using a multi-level cache strategy, including local cache (SRAM), in-memory database (DRAM), and remote storage (NOF+network), supporting preloading and smart tagging of hot data. The dynamic sharding module is configured to perform data sharding based on a consistent hashing algorithm and dynamic weight calculation; preferably, the dynamic weight calculation formula is: in, 、 and is a configurable parameter; is the CPU utilization; is I / O latency; is the memory usage; is the maximum value of CPU utilization, The maximum value of I / O delay; is the maximum value of memory usage. In the formula, normalization is used to eliminate magnitude differences and weights are calculated dynamically to improve the reliability of data sharding.

[0023] The parallel computing module is configured to coordinate computing tasks with data sharding distribution so that data can be processed directly on the storage nodes; The preloading module is configured to: analyze historical request patterns and real-time traffic characteristics based on machine learning models, predict hot data, and load it into the cache layer in advance; The resource scheduling center is configured to perform resource discovery and data transmission through a combination of CXL and RELOAD protocols. The CXL protocol is used for memory-level low-latency interaction, and the RELOAD protocol is used for cross-node routing and resource discovery.

[0024] Based on the above, the storage-computing integrated data scheduling system for high-concurrency scenarios in this embodiment improves the overall performance of the system by reducing the data transmission delay between the storage and computing units in high-concurrency scenarios. Specifically: by tightly integrating the storage unit and the computing unit, the frequent data handling overhead in the traditional system is eliminated, the data transmission path is reduced, and the I / O delay is significantly reduced; combining the CXL protocol (memory-level low-latency interaction) and NOF+ technology (SPDK+RDMA user-mode protocol stack), the cross-node communication delay is compressed to the microsecond level to meet the real-time requirements in high-concurrency scenarios. The parallel computing module realizes direct processing of data on the storage node through modular design, reduces cross-node communication overhead, and improves overall computing efficiency; based on the consistent hashing algorithm and dynamic weights, the shard size and concurrency are adjusted in real time to adapt to the load status of different nodes and avoid resource overload or idleness; based on the machine learning model to analyze historical request patterns and real-time traffic characteristics, potential hotspot data is preloaded to the cache layer in advance, reducing cold start delays during traffic peaks and improving response speed; In one embodiment, the distributed cache module includes: The level adjustment unit is configured to: automatically adjust the cache level according to the data access frequency; The cache management unit is configured to optimize cache efficiency based on a setting of a cache expiration time.

[0025] By setting a reasonable cache expiration time, you can balance cache efficiency and storage costs and prevent invalid data from occupying cache resources.

[0026] Furthermore, the preloading module is specifically configured as follows: Based on machine learning models, periodic predictions are made based on time series, analyzing periodic patterns in historical request patterns; Quantify the hotness of data based on the weight of access frequency; Burst traffic based on QPS thresholds triggers preloading at traffic peaks.

[0027] In this way, by combining LRU memory cache and persistent cache, the cache level is automatically adjusted according to the data access frequency, improving the hit rate of hot data.

[0028] In one embodiment, the steps of implementing the data sharding include: Create a new shard task and obtain the recommended shard size, recommended concurrency, and unique ID of the corresponding upload task; Check the latest upload speed data, if there is the latest upload speed data of the upload task, calculate the shard size and the number of concurrent connections as the allocation scheme, if not, use the corresponding recommended shard size and the recommended number of concurrent connections as the allocation scheme; Execute uploads according to the allocation plan and record the uploaded shard size, number of concurrent uploads, and upload time; Based on the fragment upload results, confirm whether all fragments have been uploaded successfully. If successful, merge the fragments to complete the task.

[0029] Furthermore, the data sharding implementation step further includes: Based on the multi-part upload results, confirm whether all parts have been uploaded successfully. If not, perform the following steps: A. Identify the type of failed shards, including network outages, node overloads, or storage anomalies, and record failure logs. The failure logs include the shard ID, failure time, node status, and error code for subsequent analysis and troubleshooting. B. Analyze the real-time load status of the failed node based on dynamic weight calculation to determine whether the upload failure is caused by resource overload; C. Automatically retry failed shards a preset number of times (e.g., three times to avoid excessive resource usage or an infinite retry loop). During retries, dynamically adjust the shard size or concurrency based on the current node's GPU computing task type (e.g., floating-point or integer) and load (e.g., if the node's memory usage is too high, reduce the shard size to reduce the pressure of a single transmission). D. If multiple retries fail, recalculate the dynamic weights, distribute the shards to other nodes (based on nodes with clear weights and more balanced load), and update the shard upload strategy (such as switching to low-priority shard upload mode); E. If the failed node remains unavailable (e.g., due to a network failure or hardware anomaly), the RELOAD protocol is used to query available node resources and the unfinished shards are migrated to the new node for continued upload. During the migration process, the cache consistency mechanism of the CXL protocol is used to synchronize the data status between the old and new nodes to avoid data duplication or loss. F. If the failed shard is hot data, the preloading module triggers the emergency preloading strategy, loads the relevant data into the cache layer of the backup node in advance (such as using the LRU+TTL combination solution to prioritize key data), and dynamically adjusts the preloading range based on the QPS threshold to prevent secondary failures caused by sudden traffic.

[0030] This approach significantly reduces the risk of task interruptions due to single points of failure or traffic surges through dynamic retry, migration, and preloading strategies. Combined with cache consistency protocols and compensation mechanisms, this ensures that data from failed shards is ultimately processed correctly. Furthermore, dynamic adjustments to sharding strategies and node allocation prevent resource waste or overload.

[0031] It is possible that in case of sharding failure, further steps can be taken: If both automatic retries and migrations fail, the system sends an alert to operations personnel via a pre-defined alert mechanism (e.g., email, SMS, or notifications from a monitoring platform), prompting them to manually check network, storage, or node status. The system also retains a temporary copy of the failed shard for manual analysis and decision-making regarding re-upload or rollback.

[0032] Furthermore, if a failed shard cannot be recovered through retries, the system can selectively roll back the subtask corresponding to that shard, retain the successfully uploaded shard data, and resubmit the subtask. Simultaneously, in the parallel computing module, a compensatory computing process is initiated for the computing tasks associated with the failed shard, re-executing the computing logic using redundant nodes or idle resources.

[0033] This allows the system to ensure task completion through dynamic retries, node migration, and data consistency verification (i.e., the CXL protocol) when shard uploads fail, shortening recovery time to seconds. Failures are promptly addressed to ensure stable and reliable system operation.

[0034] In one embodiment, the resource scheduling center is further configured to utilize NOF+ technology to acquire data and synchronize the results to other nodes. Thus, the CXL protocol is used to access local cache data; the RELOAD protocol queries the location of remote node resources; and the NOF+ network optimizes storage access paths, forming a comprehensive communication protocol stack. The communication protocol stack optimizations include: the CXL protocol enables low-latency memory-level interaction and supports cache coherence; the RELOAD protocol manages inter-node resource discovery and routing, supporting dynamic topology maintenance; and NOF+ technology, combined with SPDK-driven user-mode protocol stacks and RDMA, reduces I / O latency to microseconds.

[0035] In one embodiment, the parallel computing module is further configured to: In a thousand-node AB dual-plane networking architecture, computing tasks are allocated through a dynamic load balancing algorithm. The dynamic load balancing algorithm is based on distinguishing task types (e.g., intensive parallel tasks are preferentially allocated to the GPU, and logical control tasks are preferentially allocated to the CPU), and adjusting task granularity according to GPU kernel functions (e.g., splitting large tasks into subtasks that adapt to GPU kernel functions); and when executing task decomposition, cross-node resource migration is performed, and the task allocation strategy is dynamically adjusted using the RELOAD protocol to ensure maximum computing resource utilization.

[0036] Furthermore, when communicating across nodes, the resource scheduling center uses the shared memory mechanism through the CXL protocol to achieve cache consistency between nodes, reducing data replication overhead; and uses lightweight heartbeat packets through the RELOAD protocol to maintain node status, supporting second-level fault switching and dynamic topology updates.

[0037] With this, the system has AB dual-plane networking and a two-level architecture design, enabling the system to support horizontal expansion of thousands of nodes, and the dynamic load balancing algorithm ensures stable performance under large-scale deployment.

[0038] In one feasible implementation, a dynamic load balancing algorithm is employed that constructs a dynamic resource graph and combines it with adaptive entropy weighting to achieve nonlinear quantification of node load states and task allocation decisions. This algorithm adapts to complex load variations in high-concurrency scenarios through the coordinated optimization of the dynamic graph structure and entropy weighting. The dynamic load balancing algorithm specifically includes the following: (1) Node state vector Each node The status is composed of the following indicators: CPU utilization , which means real-time monitoring of CPU usage (0 means idle, 1 means fully loaded); Memory usage , indicating real-time monitoring of memory usage ratio; I / O latency , which represents the average response time of I / O requests per unit time (ms); Vector form: .

[0039] (2) Dynamic resource map Node Set , represents all available computing nodes in the system; Edge Set , represents the relationship between inter-node communication overhead and resource coordination, and is defined as: in, For nodes and The physical distance between them (e.g., number of network hops); is the attenuation coefficient, which is dynamically adjusted according to the network topology; is the communication bandwidth weight, which is obtained from the link quality evaluation of the CXL / RELOAD protocol.

[0040] (3) Adaptive entropy weight 、 and The weight of each indicator (CPU, memory, I / O) is dynamically calculated using the entropy weight method. The formula is: in, For nodes Information entropy; The entropy value of CPU utilization is calculated as follows: Analogy definition and , corresponding to the entropy values ​​of memory and I / O respectively.

[0041] (4) Node load scoring function Combining dynamic resource graph and entropy weight method to define nodes The combined load rating is: in, The CPU idleness (the higher the better for task allocation); The memory free degree (the higher the better for task allocation); I / O efficiency (the lower the latency, the better for task allocation).

[0042] (5) Task allocation strategy Use dynamic graph traversal algorithms (such as Dijkstra algorithm) to find the optimal path: Objective function: Minimize total cost in, To assign to a node Number of tasks; The constraints are: ,in, The total number of tasks, the node load score must be higher than the threshold .

[0043] The algorithm flow is: (1) Initialization phase Monitor and obtain real-time status vectors of all nodes ; Building a dynamic resource graph , calculate the communication overhead between nodes .

[0044] (2) Entropy weight calculation stage According to the current node status, calculate the entropy value of each indicator 、 and ; Dynamically update entropy weight 、 and .

[0045] (3) Load score calculation stage For each node Calculating the Composite Load Rating .

[0046] (4) Task allocation stage Using dynamic graph traversal algorithm, according to the objective function and constraints to allocate tasks.

[0047] (5) Adaptive adjustment stage Periodically update the node state vector With dynamic resource graph .

[0048] Adjust the decay coefficient in the entropy weight calculation based on task execution results (such as actual delay, resource consumption) .

[0049] The dynamic load balancing algorithm in this embodiment achieves efficient load balancing management in high-concurrency scenarios by combining a dynamic resource graph with adaptive entropy weights. Its nonlinear scoring function and dynamic graph optimization mechanism significantly outperform traditional linear weighting schemes, and it offers excellent scalability and robustness, making it suitable for scenarios with extremely high real-time and reliability requirements, such as big data analysis.

[0050] In a second aspect, this embodiment provides a storage-computing integrated data scheduling method for high-concurrency scenarios, which is applied to the storage-computing integrated data scheduling system for high-concurrency scenarios described above. The method is characterized in that it includes the following steps: System initialization: Establish a distributed storage and computing node network, where each node includes a storage unit and a computing unit; Data sharding: Dynamically distribute data to different nodes based on consistent hashing algorithm and dynamic weight calculation; Load monitoring: collects CPU utilization, memory utilization, memory bandwidth, core temperature, I / O latency, and memory usage of each node in real time; Predictive analysis: Analyze historical request patterns and real-time traffic characteristics based on machine learning models to predict hotspot data; Preload execution: Based on the prediction results of hot data, the hot data is loaded into the cache layer of the corresponding node in advance; Task allocation: Coordinate computing tasks with data sharding allocation, allowing data to be processed directly on storage nodes; Result merging: Collect the calculation results of each node, merge them into the final result and return it to the user.

[0051] It should be noted that the storage-computing integrated data scheduling method for high-concurrency scenarios (hereinafter referred to as: this method) of this embodiment corresponds to the aforementioned storage-computing integrated data scheduling system for high-concurrency scenarios (hereinafter referred to as: this system). Therefore, the content not specifically described in this method (including but not limited to technical implementation means, technical effects, etc.) can be referred to the relevant description in this system, and this text will not go into details here.

[0052] To sum up, the storage and computing integrated data scheduling system and method for high-concurrency scenarios in this embodiment, through storage and computing fusion architecture, dynamic resource optimization, intelligent cache preloading and protocol collaborative innovation, has built a set of efficient data scheduling technologies for high-concurrency scenarios, solving the pain points of traditional systems in terms of latency, resource utilization, scalability and reliability.

[0053] In the embodiments provided herein, it should be understood that the embodiments described herein can be implemented using hardware, software, firmware, middleware, code, or any appropriate combination thereof. For hardware implementation, the processor may be implemented in one or more of the following: an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, or other electronic units designed to implement the functionality described herein, or any combination thereof. For software implementation, some or all of the processes of the embodiments may be performed by a computer program instructing the relevant hardware. During implementation, the program may be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media includes any medium that facilitates the transmission of a computer program from one location to another. The storage medium may be any available medium that can be accessed by a computer. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing the desired program code in the form of instructions or data structures and accessible by a computer.

[0054] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent replacements for some of the technical features therein. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A storage and computing integrated data scheduling system for high-concurrency scenarios, characterized by: include: The storage and computing fusion architecture module is configured to: couple the data storage unit with the computing unit to enable the coordinated operation of the data storage function and the computing function; The distributed cache module is configured to cache data using a multi-level cache strategy, wherein the multi-level cache strategy includes: local cache, memory database and remote storage; The dynamic sharding module is configured to perform data sharding based on consistent hashing and dynamic weight calculation; The parallel computing module is configured to coordinate computing tasks with data sharding distribution so that data can be processed directly on the storage nodes; The preloading module is configured to: analyze historical request patterns and real-time traffic characteristics based on machine learning models, predict hot data, and load it into the cache layer in advance; The resource scheduling center is configured to perform resource discovery and data transmission through a combination of CXL and RELOAD protocols. The CXL protocol is used for memory-level low-latency interaction, and the RELOAD protocol is used for cross-node routing and resource discovery.

2. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 1 is characterized in that: The distributed cache module includes: The level adjustment unit is configured to: automatically adjust the cache level according to the data access frequency; The cache management unit is configured to optimize cache efficiency based on a setting of a cache expiration time.

3. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 1 is characterized in that: The preloading module is specifically configured as follows: Based on machine learning models, periodic predictions are made based on time series, analyzing periodic patterns in historical request patterns; Quantify the hotness of data based on the weight of access frequency; Burst traffic based on QPS thresholds triggers preloading at traffic peaks.

4. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 1 or 3 is characterized in that: The steps of implementing the data sharding include: Create a new shard task and obtain the recommended shard size, recommended concurrency, and unique ID of the corresponding upload task; Check the latest upload speed data, if there is the latest upload speed data of the upload task, calculate the shard size and the number of concurrent connections as the allocation scheme, if not, use the corresponding recommended shard size and the recommended number of concurrent connections as the allocation scheme; Execute uploads according to the allocation plan and record the uploaded shard size, number of concurrent uploads, and upload time; Based on the fragment upload results, confirm whether all fragments have been uploaded successfully. If successful, merge the fragments to complete the task.

5. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 4 is characterized in that: The data sharding implementation step further includes: Based on the multi-part upload results, confirm whether all parts have been uploaded successfully. If not, perform the following steps: A. Identify the type of failed shards, including network outage, node overload, or storage anomaly, and record a failure log, including the shard ID, failure time, node status, and error code. B. Analyze the real-time load status of the failed node based on dynamic weight calculation to determine whether the upload failure is caused by resource overload; C. Automatically retry the failed shards a preset number of times, dynamically adjusting the shard size or concurrency based on the computing task type and load of the current node GPU during the retry. D. If multiple retries fail, recalculate the dynamic weight, distribute the shards to other nodes, and update the shard upload strategy; E. If the failed node remains unavailable, the RELOAD protocol is used to query available node resources and migrate the unfinished shards to the new node for continued upload. During the migration process, the cache consistency mechanism of the CXL protocol is used to synchronize the data status between the old and new nodes. F. If the failed shard is hot data, the preloading module triggers the emergency preloading strategy, loads the relevant data into the cache layer of the standby node in advance, and dynamically adjusts the preloading range in combination with the QPS threshold.

6. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 1 is characterized in that: The remote storage is a NOF+ network.

7. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 6 is characterized in that: The resource scheduling center is further configured to: acquire data using NOF+ technology and synchronize the results to other nodes.

8. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 1 is characterized in that: The parallel computing module is further configured to: Under the AB dual-plane networking architecture with a scale of thousands of nodes, computing tasks are allocated through a dynamic load balancing algorithm, and the task granularity is adjusted according to the execution efficiency of GPU kernel functions; and when executing task decomposition, cross-node resource migration is performed, and the task allocation strategy is dynamically adjusted using the RELOAD protocol.

9. The storage and computing integrated data scheduling system for high-concurrency scenarios according to claim 8 is characterized in that: When communicating across nodes, the resource scheduling center uses a shared memory mechanism via the CXL protocol to achieve cache consistency between nodes; and uses a lightweight heartbeat packet via the RELOAD protocol to maintain node status.

10. A storage-computing integrated data scheduling method for high-concurrency scenarios, applied to the storage-computing integrated data scheduling system for high-concurrency scenarios as described in any one of claims 1-9, characterized in that: The method comprises the following steps: System initialization: Establish a distributed storage and computing node network, where each node includes a storage unit and a computing unit; Data sharding: Dynamically distribute data to different nodes based on consistent hashing algorithm and dynamic weight calculation; Load monitoring: collects CPU utilization, memory utilization, memory bandwidth, core temperature, I / O latency, and memory usage of each node in real time; Predictive analysis: Analyze historical request patterns and real-time traffic characteristics based on machine learning models to predict hotspot data; Preload execution: Based on the prediction results of hot data, the hot data is loaded into the cache layer of the corresponding node in advance; Task allocation: Coordinate computing tasks with data sharding allocation, allowing data to be processed directly on storage nodes; Result merging: Collect the calculation results of each node, merge them into the final result and return it to the user.

Images