Data interaction method, platform, device, medium and program product

Abstract

The application relates to the technical field of data processing, and provides a data interaction method, platform, device, medium and program product, wherein the method comprises the following steps: receiving a transmission request of sending data from the source device to the target device; in response to the transmission request, reading to-be-transmitted data in the source device through a first working thread, and writing the to-be-transmitted data into a shared storage space; reading the to-be-transmitted data in the shared storage space through a second working thread, and writing the to-be-transmitted data into the target device; wherein the first working thread and the second working thread are executed in parallel in a pipeline mode. Through the introduction of the shared storage space in the host device and the cooperation of the double-thread pipeline parallel processing mechanism, the software and hardware decoupling and high-speed concurrent transmission of data transmission between heterogeneous devices are realized, and the application has the advantages of strong universality, high transmission efficiency and good system compatibility.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data interaction method, platform, device, medium, and program product. Background Technology

[0002] With the evolution of the Artificial Intelligence Radio Access Network (AI-RAN) architecture, the use of heterogeneous computing platforms such as Field Programmable Gate Arrays (FPGAs) and General-purpose computing on Graphics Processing Units (GPGPUs) to collaboratively process radio access network and artificial intelligence services has become a mainstream trend.

[0003] Currently, to achieve data transfer between devices, Direct Memory Access (DMA) technology is commonly used to enable direct interconnection between devices. However, this technology heavily relies on specific hardware designs and proprietary data transfer protocols, resulting in poor versatility. Furthermore, developers often need to perform targeted low-level adaptations and repetitive development for different hardware platforms, leading to high development costs, high hardware-software coupling, and poor portability in existing solutions. Summary of the Invention

[0004] This invention provides a data interaction method, platform, device, medium, and program product to solve the defects in related technologies where data transmission between heterogeneous devices relies excessively on specific hardware and protocols, resulting in poor compatibility and difficulty in porting.

[0005] This invention provides a data interaction method applied to a master control device. The master control device has a shared storage space and is communicatively connected to a source device and a target device, which are heterogeneous devices. The method includes:

[0006] Receive a data transmission request from the source device to the target device;

[0007] In response to the transmission request, the first working thread reads the data to be transmitted from the source device and writes the data to be transmitted into the shared storage space;

[0008] The second working thread reads the data to be transmitted from the shared storage space and writes the data to be transmitted into the target device.

[0009] The first worker thread and the second worker thread are executed in parallel in a pipeline manner.

[0010] According to a data interaction method provided by the present invention, the step of reading the data to be transmitted from the source device through a first working thread further includes:

[0011] According to the transmission request, the data to be transmitted is divided into multiple data packets, so that the first working thread and the second working thread can perform data reading and writing operations at the granularity of the data packets.

[0012] According to a data interaction method provided by the present invention, the step of dividing the data to be transmitted into multiple data packets according to the transmission request includes:

[0013] The transmission request is parsed to obtain the total data volume and target address of the data to be transmitted;

[0014] The data size of a single data packet is determined based on a preset slicing strategy, and the data to be transmitted is divided into multiple data segments according to the total data volume and the data size.

[0015] Generate a data packet containing a metadata header for each data segment. The metadata header includes the packet sequence number of the data packet and the destination address.

[0016] According to a data interaction method provided by the present invention, the step of reading the data to be transmitted from the source device through a first working thread and writing the data to be transmitted into the shared storage space includes:

[0017] The segmented data packets are added sequentially to the first task queue;

[0018] The first working thread extracts the first data packet currently to be processed from the first task queue.

[0019] Based on the information in the first data packet, the target data corresponding to the first data packet is read from the source device and written into the shared storage space.

[0020] According to a data interaction method provided by the present invention, the step of retrieving the first data packet to be processed from the first task queue through the first working thread includes:

[0021] While the data to be transmitted is divided into multiple data packets, the first task queue is monitored by the first working thread;

[0022] If a data packet is detected in the first task queue, the first data packet is extracted from the first task queue according to the first-in-first-out principle.

[0023] According to a data interaction method provided by the present invention, after the step of writing the target data into the shared storage space, the method further includes:

[0024] Generate a second data packet carrying the storage address of the target data in the shared storage space, and add the second data packet to the second task queue;

[0025] The step of reading the data to be transmitted from the shared storage space and writing the data to be transmitted to the target device through the second working thread includes:

[0026] The second data packet is extracted from the second task queue using the second worker thread;

[0027] Based on the storage address carried in the second data packet, the target data corresponding to the storage address is read from the shared storage space and written to the target device.

[0028] According to a data interaction method provided by the present invention, after the step of writing the target data into the target device, the method further includes:

[0029] The second working thread destroys the second data packet in the second task queue and releases the memory resources occupied by the target data in the shared storage space.

[0030] A data interaction method provided by the present invention further includes:

[0031] Get the current system load status;

[0032] Based on the system load status, adjust the data size of a single data packet in the preset slicing strategy, as well as the queue depth of the first and second task queues.

[0033] The present invention also provides a data interaction device, which is applied to a master control device. The master control device is provided with a shared storage space and is communicatively connected to a source device and a target device, respectively. The source device and the target device are heterogeneous devices. The device includes:

[0034] A request receiving unit is configured to receive a transmission request for sending data from the source device to the target device;

[0035] The first copy unit is configured to, in response to the transmission request, read the data to be transmitted from the source device through a first working thread and write the data to be transmitted into the shared storage space;

[0036] The second copy unit is used to read the data to be transmitted from the shared storage space through the second working thread, and write the data to be transmitted into the target device;

[0037] The first worker thread and the second worker thread are executed in parallel in a pipeline manner.

[0038] The present invention also provides a data interaction platform, including a master control device, at least one source device and at least one target device, wherein the master control device is provided with a shared storage space, and the master control device is communicatively connected to the at least one source device and the at least one target device respectively, and the master control device is used to implement the data interaction method as described above.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the data interaction method as described above.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data interaction method as described above.

[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data interaction method as described above.

[0042] The data interaction method, platform, device, medium, and program products provided by this invention construct a data transfer mechanism based on an intermediate buffer by setting up a shared storage space on the main control device and using a first working thread and a second working thread on the main control device to perform copy operations from the source device to the shared storage and from the shared storage to the target device, respectively. This mechanism first achieves logical decoupling between the source and target devices through the shared storage space, enabling data interaction between heterogeneous devices without establishing direct hardware connections or following specific point-to-point transmission protocols, thus overcoming the compatibility challenges caused by differences in underlying hardware. Furthermore, by controlling the first and second working threads to execute in parallel in a pipelined manner, the system can simultaneously write the first part of data to the target device and read the next part of data from the source device. This parallel processing mode effectively masks the input / output latency of a single device and fully utilizes the bandwidth resources of the system bus, thereby significantly improving the throughput and overall efficiency of data transmission between heterogeneous devices while ensuring transmission stability. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the data interaction method provided by the present invention;

[0045] Figure 2 This is a schematic diagram of the FPGA and GPU interconnection architecture provided by the present invention;

[0046] Figure 3 This is a schematic diagram of the data interaction pipeline between the FPGA and the GPU provided by the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of the data interaction device provided by the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] With the rapid development of communication technology, the 6G AI-RAN architecture was designed from the outset with a strong focus on the deep integration of artificial intelligence (AI) technology. To simultaneously support Radio Access Network (RAN) services and AI services on a single hardware device, a heterogeneous computing platform composed of different types of computing units, such as FPGAs and GPGPUs or Graphics Processing Units (GPUs), is typically required. In practical applications, achieving efficient and stable data interaction between these heterogeneous devices is crucial to realizing the performance of this architecture.

[0051] In related technologies, Direct Memory Access (DMA) technology is commonly used to achieve direct interconnection between devices for data transfer. This approach allows hardware subsystems to directly access system memory, improving data transfer efficiency to some extent. However, this traditional DMA-based direct interconnection method has certain limitations and drawbacks in practical applications:

[0052] First, this approach is heavily reliant on specific device hardware designs and proprietary data transfer protocols. If the device itself does not support a particular DMA technology, interconnection is impossible, which severely limits the flexibility of hardware selection.

[0053] Secondly, developers need to perform targeted development and underlying adaptation for different hardware platforms, which means that a lot of manpower needs to be invested in redevelopment whenever the hardware is changed or the system is upgraded.

[0054] Finally, due to the strong coupling with specific hardware and protocols, the existing solutions have poor portability, making it difficult to build a universal data interaction mechanism that can adapt to a variety of heterogeneous device combinations.

[0055] To address this issue, this invention provides a universal data interaction method and platform. Utilizing a master control device as a relay hub, and through the cooperation of a first and second working thread, data interaction between heterogeneous devices is achieved. This method decouples the source and target devices, eliminating reliance on direct hardware interconnection protocols (such as DMA) between devices. This results in strong versatility and hardware compatibility; regardless of the heterogeneous device's architecture, interconnection can be achieved as long as it can communicate with the master control device. Furthermore, although the dual-threaded pipelined parallel execution mechanism adds a relay step via the master control device, it maximizes the utilization of uplink and downlink bus bandwidth, ensuring high data transmission throughput and avoiding performance bottlenecks caused by serial processing, thus overcoming the aforementioned shortcomings.

[0056] Figure 1 This is a flowchart illustrating the data interaction method provided by the present invention, as shown below. Figure 1 As shown, the method is applied to a master control device, which has a shared storage space and is communicatively connected to a source device and a target device, which are heterogeneous devices.

[0057] It should be noted that the method provided in this embodiment of the invention mainly runs in a heterogeneous computing platform architecture. The core of this architecture is to solve the problems of difficult data interaction between hardware devices with different architectures and reliance on specific hardware protocols (such as DMA).

[0058] The execution entity of the data interaction method in this embodiment of the invention is a master control device. This master control device can be any form of computing device, such as a server, personal computer, industrial control computer, or embedded system, and its core includes one or more central processing units (CPUs) and main memory. The master control device is equipped with a shared storage space, which can be the system memory of the master control device (such as shared memory), used as a temporary relay station during data interaction. The master control device has the ability to communicate with external devices; for example, it can connect to various hardware devices via PCIe (Peripheral Component Interconnect Express) bus, Ethernet, or other high-speed communication interfaces. In this embodiment of the invention, the core role of the master control device is as a data relay and scheduling center. Through its powerful task processing and memory management capabilities, it decouples data dependencies between heterogeneous devices, realizing a universal data interaction platform. By utilizing the advantages of the CPU in processing complex logic and the high-speed read / write characteristics of memory, the master control device can effectively manage and schedule data flows.

[0059] In this embodiment of the invention, the source device and the target device are heterogeneous devices that communicate with the host device. The source device is the initiator of data transmission, while the target device is the receiver of data transmission. It should be understood that source and target are relative concepts; hardware that acts as the source device in one data transmission task may also act as the target device in another task. These two devices establish communication connections with the host device, for example, via a PCIe bus.

[0060] Heterogeneous devices refer to devices whose hardware architecture, instruction set, operating principle, or manufacture differs from each other, making efficient direct data exchange impossible. For example, the source device might be an FPGA, while the target device is a GPGPU or GPU, or vice versa. In applications such as 6G AI-RAN, these heterogeneous devices are typically high-performance hardware accelerators, such as FPGAs for signal processing and GPUs for artificial intelligence computing. Their architectures are vastly different, creating barriers to direct communication. This invention aims to solve the data interaction challenges between these heterogeneous devices.

[0061] This invention provides a data interaction method, primarily applicable to heterogeneous computing platforms, especially in scenarios requiring decoupling of different hardware devices to achieve universal and efficient data exchange, such as data interaction between FPGA and GPU in a 6G AI-RAN architecture. The specific process of this method includes the following steps:

[0062] Step S10: Receive a transmission request to send data from the source device to the target device.

[0063] Specifically, the method of this invention is applied to a master control device. A memory region is pre-configured or dynamically partitioned on the master control device as a shared storage space. This shared storage space can physically be part of the master control device's main memory, and logically serves as a data buffer for temporarily storing data to be transmitted between the source and target devices. For example, it can be implemented as a First-In-First-Out (FIFO) queue. Simultaneously, the master control device establishes communication connections with both the source and target devices via a communication bus (e.g., a PCIe bus). In this embodiment of the invention, the source and target devices are heterogeneous devices; for example, the source device can be an FPGA card, while the target device can be a GPU accelerator card.

[0064] First, the master device receives a data transmission request from the source device to the target device. Here, the transmission request is an instruction or signal indicating that the source device (e.g., an FPGA) has a batch of data to send to the target device (e.g., a GPU) for further processing. This transmission request can be generated by an upper-layer application running on the master device, or it can be notified by the source device via an interrupt or other means to initiate a data transmission task from the source device to the target device. The transmission request typically contains necessary information, such as the starting address of the data to be transmitted in the source device, the total amount of data, and the address information of the target device to which the data needs to be sent.

[0065] Step S20: In response to the transmission request, the data to be transmitted in the source device is read through the first working thread, and the data to be transmitted is written into the shared storage space.

[0066] Specifically, upon receiving a transmission request, the central processing unit (CPU) of the master device starts or assigns a dedicated CPU thread, the first worker thread, to handle the data copying task from the source device to the master device. This first worker thread can be understood as a producer thread. Based on the information in the transmission request, it actively reads the data to be transmitted from the internal memory of the source device via the communication bus (such as the PCIe bus) between the master device and the source device, and then writes this data into the shared storage space (such as shared memory) of the master device. For example, this thread reads a data block from the onboard memory of the FPGA and stores it in the CPU's main memory buffer. In this process, the data to be transmitted completes the first traversal from the source device to the master device.

[0067] Step S30: The second working thread reads the data to be transmitted from the shared storage space and writes the data to be transmitted into the target device.

[0068] The first worker thread and the second worker thread are executed in parallel in a pipeline manner.

[0069] Specifically, similar to the first worker thread, the CPU of the master device starts or assigns another dedicated CPU thread, the second worker thread, to handle the data copying task from the master device to the target device. This second worker thread can be understood as a consumer thread, responsible for monitoring the shared memory space. Once data is detected in the shared memory space, it reads the data to be transferred and, based on the target address information in the transfer request, writes this data into the internal memory of the target device via the communication bus between the master device and the target device. For example, this thread reads data blocks from the CPU's main memory buffer and writes them into the GPU's video memory. In this process, the data to be transferred completes its second traversal from the master device to the target device, thus ultimately realizing the data migration from the source device to the target device.

[0070] It is understandable that the first and second worker threads execute in parallel using a pipelined approach. Pipeline parallel execution means that the first and second worker threads work collaboratively as producers and consumers, with their execution overlapping in time. Specifically, while the second worker thread (consumer) is reading the Nth data block from shared memory and sending it to the target device, the first worker thread (producer) can simultaneously read the N+1th data block from the source device and write it to shared memory. This parallel pipelined operation mode avoids the bus idleness and waiting caused by the traditional serial mode of complete read followed by complete write. It allows the communication bus from the source device to the master device and the communication bus from the master device to the target device to be active as simultaneously as possible, thereby greatly improving the overall data transmission throughput and fully utilizing the system bus bandwidth.

[0071] For example, Figure 2 This is a schematic diagram of the FPGA and GPU interconnection architecture provided by the present invention, as shown below. Figure 2 As shown, to better understand the application of the solution of this invention in a real heterogeneous computing scenario, the following explanation uses a typical data processing flow in a 6G AI-RAN (Artificial Intelligence Radio Access Network) system as an example. In this application scenario, the main control device can be a server host containing a CPU, the source device is specifically an FPGA accelerator card used to perform preprocessing of physical layer signals, and the target device is specifically a GPU accelerator card used to perform subsequent AI inference tasks. The specific process is as follows:

[0072] After the FPGA completes the preprocessing of the signal data, it generates a transmission request to send data to the GPU. The CPU responds to this request, using its powerful logic control capabilities as an intermediary. Thread A (i.e., the first working thread), which is started or scheduled internally by the CPU, acts as the producer thread and reads the processed data to be transmitted from the FPGA via the PCIe bus.

[0073] Thread A writes the data it reads to the shared memory region (i.e., shared storage space) in the CPU system memory. For example... Figure 2 As shown, in order to better manage the data flow, this shared storage space can be managed in conjunction with a queue mechanism. Thread A enqueues the data to ensure the orderliness of the storage.

[0074] Meanwhile, thread B (the second worker thread) inside the CPU executes in parallel with thread A as a consumer thread. When thread B detects that there is data to be transferred in the shared memory, it immediately reads the data from the shared memory and writes it to the GPU's video memory via the PCIe bus for the GPU to perform subsequent AI model inference.

[0075] In this pipelined parallel mode, while thread A is moving the (N+1)th block of data from the FPGA, thread B is moving the Nth block of data to the GPU. This approach not only avoids the problem of the lack of a direct hardware path between the FPGA and the GPU, but also masks the latency of memory copying through the dual-threaded pipeline of the CPU, ensuring efficient data flow in the heterogeneous computing platform.

[0076] The method provided in this invention constructs a data transfer mechanism based on an intermediate buffer by setting up a shared storage space on the master control device and using a first working thread and a second working thread on the master control device to perform copy operations from the source device to the shared storage and from the shared storage to the target device, respectively. This mechanism first achieves logical decoupling between the source and target devices through the shared storage space, enabling data interaction between heterogeneous devices without establishing direct hardware connections or following specific point-to-point transmission protocols, thus overcoming compatibility challenges caused by differences in underlying hardware. Furthermore, by controlling the first and second working threads to execute in parallel in a pipelined manner, the system can simultaneously write the first part of data to the target device and read the next part of data from the source device. This parallel processing mode effectively masks the input / output latency of a single device and fully utilizes the bandwidth resources of the system bus, thereby significantly improving the throughput and overall efficiency of data transmission between heterogeneous devices while ensuring transmission stability.

[0077] Based on any of the above embodiments, before step S20, the method further includes:

[0078] Step S20': According to the transmission request, the data to be transmitted is divided into multiple data packets, so that the first working thread and the second working thread can perform data reading and writing operations at the granularity of the data packets.

[0079] It should be noted that in heterogeneous computing scenarios, such as 6G AI-RAN systems, the amount of data exchanged between source and target devices is typically enormous, such as high-sampling-rate radio frequency signal data or weight parameters of large-scale AI models. If all data is transferred at once, it would lead to excessively long pipeline wait times. Therefore, this embodiment of the invention introduces a data fragmentation mechanism.

[0080] Specifically, upon receiving a transmission request, the master control device first evaluates the data task indicated by the request. To achieve efficient pipeline parallelism, a large data transmission task is broken down into several small, independently operable task units.

[0081] Specifically, the master control device logically divides the originally continuous data to be transmitted into several smaller segments according to certain rules. These segments constitute data packets. Here, data packets represent the basic granularity of memory management and thread scheduling within the master control device.

[0082] It is understandable that using data packets as the granularity means that when the first worker thread reads data, it does not read all the data at once, but reads only a data packet size at a time and puts it into the shared storage space; similarly, the second worker thread also takes out a data packet size from the shared storage space and writes it to the target device at a time.

[0083] This packet-splitting mechanism breaks down the workload into smaller, manageable parts. Once the first worker thread finishes reading the first data packet and placing it in shared memory, it immediately moves on to reading the second data packet. Simultaneously, since the first data packet already exists in shared memory, the second worker thread can start immediately and write it to the target device. This mechanism ensures that the first and second worker threads overlap to the greatest extent possible on the timeline, thus truly establishing a highly efficient pipeline.

[0084] This invention, through segmenting the data to be transmitted into multiple data packets and using these data packets as the basic granularity for thread read / write operations, enables the data transmission process to be pipelined at a finer level. This not only avoids thread idleness caused by waiting for large data blocks to complete, greatly improving the system's concurrent processing capabilities and bus utilization, but also significantly reduces transmission latency, allowing the target device to receive partial data and begin subsequent processing more quickly.

[0085] Based on any of the above embodiments, in order to ensure that the data can still be correctly reassembled and transmitted to the accurate location after being segmented, fine-grained control of the packet segmentation process is required. Therefore, step S20' specifically includes:

[0086] The transmission request is parsed to obtain the total data volume and target address of the data to be transmitted;

[0087] The data size of a single data packet is determined based on a preset slicing strategy, and the data to be transmitted is divided into multiple data segments according to the total data volume and the data size.

[0088] Generate a data packet containing a metadata header for each data segment. The metadata header includes the packet sequence number of the data packet and the destination address.

[0089] Specifically, the transmission request carries key parameters for this interaction. The master device first parses the request, extracting the total amount of data to be transmitted (e.g., 100MB of data to be transmitted in this case) and the target address of the data on the target device (i.e., the starting physical or logical address where the data will ultimately be written to the GPU memory or FPGA storage space). Obtaining this information is the foundation for subsequent slice calculations.

[0090] In addition, the host device has a pre-defined slicing strategy. This strategy is usually formulated based on hardware parameters such as the host device's CPU cache size, maximum PCIe bus load, and shared memory page size. For example, the strategy might specify that the data size of a single data packet is 4MB to balance the overhead of context switching with pipeline smoothness.

[0091] Based on the determined data size (e.g., 4MB) and the total amount of data acquired (e.g., 100MB), the main control device calculates the number of segments to be divided (e.g., 100 / 4=25), thus logically dividing the original data to be transmitted into multiple consecutive data segments. These data segments constitute the net payload of the original data.

[0092] Plain data segments alone are insufficient. To ensure orderly transmission and accurate addressing, an identifier needs to be added to each data segment. Therefore, the master control device generates a metadata header for each segmented data segment and combines this header with the data segment to form the final data packet.

[0093] Understandably, the metadata header contains at least two key pieces of information: the packet sequence number and the destination address. The packet sequence number identifies the order of the data packet within the original data, for example, packet number 1, packet number 2, and so on. This is crucial for the target device to reassemble data at the receiving end or to perform verification in the event of out-of-order transmission. The destination address refers to the specific location within the target device where this particular data packet should be stored. Typically, the destination address of the first data packet is the starting address obtained after parsing the transmission request, while the destination addresses of subsequent data packets are calculated based on the packet sequence number and data size using an offset. This way, when the second worker thread receives a data packet, it can directly read the header to determine which address on the target device to write it to, without needing to query the global information again.

[0094] This invention, through parsing requests and slicing data packets based on a preset strategy, adapts the packet size to the hardware environment (such as cache and bus characteristics), thereby achieving optimal transmission performance. Simultaneously, by adding a metadata header containing the packet sequence number and destination address to the data packet, self-descriptive data is achieved, ensuring the sequential traceability and addressing accuracy of data during pipelined transmission, avoiding the risk of data misalignment or loss, and enhancing the system's robustness.

[0095] Based on any of the above embodiments, step S20 specifically includes:

[0096] Step S21: Add the segmented data packets to the first task queue in sequence;

[0097] Step S22: Extract the first data packet to be processed from the first task queue through the first working thread;

[0098] Step S23: Based on the information of the first data packet, read the target data corresponding to the first data packet from the source device and write the target data into the shared storage space;

[0099] It should be noted that the above embodiments described the segmentation of data into data packets, while the embodiments of the present invention will further explain how these data packets are processed in an orderly manner by the first worker thread. The embodiments of the present invention introduce a task queue to decouple task generation from task execution.

[0100] Specifically, a first-in-first-out (FIFO) data structure, namely the first task queue, is pre-allocated and maintained in the memory of the main control device. This queue is specifically used to temporarily store read tasks that are yet to be processed.

[0101] When the scheduling logic (or main control thread) of the master control device cuts the data into data packets (i.e., task units containing metadata headers) according to the transmission request, it does not directly call the underlying hardware interface, but instead pushes these data packets into the tail of the first task queue as task descriptors.

[0102] Understandably, the data packets added to the queue at this point typically refer to structure objects containing control information such as address, length, and sequence number, rather than the original business data itself. This significantly reduces the memory bandwidth usage of queue operations, enabling lightweight task scheduling.

[0103] The first worker thread, acting as a consumer, is designed to continuously poll or block while monitoring the status of the first task queue. When a task is in the queue, the first worker thread pops a data packet from the head of the queue as the first data packet to be processed. This "first data packet" does not specifically refer to the first one in the sequence, but rather to the data packet object acquired by the thread in the current processing loop. In this way, the first worker thread always processes the first task to arrive in the queue, ensuring processing order.

[0104] The first worker thread parses the metadata (such as target address, data size, etc.) from the extracted first data packet. Then, it uses this information to drive the communication interface (such as a PCIe interface) between the master device and the source device, initiating a read operation command. The source device responds to the command, sending the actual payload data, i.e., the target data, to the master device. The first worker thread receives this target data and writes it to a designated location in the pre-allocated shared storage space of the master device. Only then is a complete data packet reading task truly finished. This step realizes the transformation from task logic to physical data transfer.

[0105] This invention, through the introduction of a first task queue as a buffer pool, achieves asynchronous decoupling between data sharding logic and data reading logic. Even if the sharding speed and reading speed are mismatched, the task queue can still act as a buffer, preventing thread blocking or resource contention. Simultaneously, through a clear task extraction and execution mechanism, the master control device can read data from the source device at an extremely stable pace, laying a solid foundation for subsequent pipeline processing.

[0106] Based on any of the above embodiments, in order to further reduce latency, it is necessary to eliminate the time gap between task generation and task execution. Therefore, step S22 specifically includes:

[0107] Step S221: While dividing the data to be transmitted into multiple data packets, the first task queue is monitored through the first working thread;

[0108] Step S222: If a data packet is detected in the first task queue, the first data packet is extracted from the first task queue according to the first-in-first-out principle.

[0109] Specifically, the main logic of the master control device (i.e., the master control thread) is performing data segmentation and enqueueing operations based on the transmission requests. At the same time, the first worker thread is not in a dormant state waiting for all segmentation work to be completed, but is in an active state, monitoring the changes in the first task queue in real time.

[0110] This design avoids the initial waiting delay caused by processing all data packets after full segmentation. It can be understood as follows: while the main control device is segmenting and generating data packets, the first worker thread is monitoring the first task queue. Once a data packet is generated and placed in the queue, the first worker thread can immediately see it.

[0111] When the monitoring mechanism detects that the first task queue is not empty, the first worker thread immediately triggers the extraction action, retrieving the corresponding data packets from the queue according to the first-in, first-out (FIFO) principle. Adhering to the FIFO principle is crucial here because the data sent by the source device typically has a strict logical order (such as the frame order of a video stream or the layer weight order of a neural network). By strictly executing FIFO extraction, the order in which the first worker thread reads data remains highly consistent with the order in which the master control device segments the data, thus ensuring that the data stream maintains the correct logical sequence when entering the shared storage space, eliminating the need for subsequent complex reordering operations.

[0112] This invention eliminates the idle waiting time during task startup by employing a parallel mechanism of packet-by-packet monitoring. The first worker thread can begin working immediately upon the generation of the first data packet, significantly improving the system's response speed. Simultaneously, strictly adhering to the first-in, first-out (FIFO) principle ensures the orderly transmission of data, guaranteeing data integrity and logical correctness during high-speed data flow between heterogeneous devices, and reducing system error correction costs.

[0113] Based on any of the above embodiments, after step S23, the method further includes:

[0114] Generate a second data packet carrying the storage address of the target data in the shared storage space, and add the second data packet to the second task queue;

[0115] Accordingly, step S30 specifically includes:

[0116] Step S31: Extract the second data packet from the second task queue using the second working thread;

[0117] Step S32: Based on the storage address carried in the second data packet, read the target data corresponding to the storage address from the shared storage space and write the target data into the target device.

[0118] Specifically, after the first worker thread successfully writes the target data read from the source device into the shared storage space, it needs to send a completion signal to the downstream. In the actual implementation, the first worker thread (or the relevant callback function) generates a new task descriptor, namely the second data packet. Unlike the first data packet, the key payload of this second data packet has changed. It not only contains the size of the data and the final destination address of the target device, but more importantly, it carries the current storage address of this data in the master device's memory (i.e., a pointer or offset in the shared storage space).

[0119] Subsequently, the first worker thread adds this generated second data packet to the second task queue. This queue serves as a bridge connecting the first and second worker threads.

[0120] Similar to the first worker thread, the second worker thread constantly monitors the second task queue. Once a new element is detected in the queue, the second worker thread immediately retrieves the second data packet from the head of the queue.

[0121] Understandably, the above mechanism ensures a strict causal relationship: the second worker thread will only be notified once the data has been safely and securely placed in the shared storage space. This prevents the second worker thread from reading a memory region that is not yet ready, thus avoiding reading dirty or empty data.

[0122] After receiving the second data packet, the second worker thread parses its metadata. By reading the storage address carried within the packet, the second worker thread can accurately locate the target data temporarily stored in the shared memory space. Subsequently, the second worker thread calls the communication interface between the master device and the target device (such as a PCIe write command) to write the data read from the master device's shared memory to the specified location on the target device. Thus, the data completes the journey from the master device's memory to the target device's hardware.

[0123] This invention establishes a second task queue, creating a reliable synchronization handshake mechanism between the first and second worker threads. Tasks are only enqueued after the first worker thread has completed data storage, ensuring the timing correctness of each stage of the pipeline. Simultaneously, by transmitting the storage address in the shared memory space within the second data packet, zero-copy data transfer logic in memory is achieved. That is, the data itself remains stationary in memory; only the address pointer is transmitted. This significantly reduces the CPU's memory copy overhead and improves the overall efficiency of data interaction.

[0124] Based on any of the above embodiments, after step S32, the method further includes:

[0125] The second working thread destroys the second data packet in the second task queue and releases the memory resources occupied by the target data in the shared storage space.

[0126] Specifically, once the second worker thread confirms that the data has been successfully written to the target device (e.g., by receiving a write completion confirmation from the PCIe controller, or by default writing success according to the protocol), the historical mission of the data packet in the master device is over.

[0127] At this point, the second worker thread (or a dedicated resource reclamation thread) performs cleanup operations: First, the second data packet is destroyed. This means removing the task descriptor object from memory and completely removing the second data packet from the system task list to prevent duplicate processing. Second, and most importantly, the memory resources occupied by the target data in the shared storage space are released. The master device marks the memory block originally allocated to this data as free, returning it to the system's memory pool.

[0128] In this way, this memory can be reused for subsequent new transmission requests (such as the next batch of data from the source device). The entire shared storage space actually acts as a circular buffer, with data constantly flowing in, being temporarily stored, flowing out, and being destroyed in a continuous cycle.

[0129] This invention establishes a complete closed-loop data lifecycle management mechanism. By promptly destroying task packets and releasing memory resources after transmission, it effectively prevents memory leaks and ensures that the memory usage of the main control device remains at a stable level during long-term, high-load data interaction. This is crucial for ensuring system robustness and long-term operational stability, enabling limited physical memory to support unlimited data stream transmission.

[0130] Based on any of the above embodiments Figure 3 This is a schematic diagram of the data interaction pipeline between the FPGA and GPU provided by the present invention, as shown below. Figure 3 As shown, the entire system consists of a CPU (main control device) located at the center, and FPGAs and GPUs located on both sides. In this embodiment of the invention, the FPGA can be set as the source device (e.g., responsible for front-end signal acquisition and preprocessing), while the GPU can be set as the target device (e.g., responsible for back-end artificial intelligence inference calculations). Of course, the GPU can also be set as the source device, and the FPGA as the target device. To avoid redundant description, this embodiment of the invention will mainly be described with the FPGA as the source device and the GPU as the target device.

[0131] The CPU acts as the core hub, connecting to both the FPGA and GPU via a communication bus (such as the PCIe bus). Internally, the CPU contains shared memory and a first and second working thread for executing logic.

[0132] First, please pay attention to the upward arrow pointing from the FPGA to the CPU on the left side of the diagram, and the corresponding white squares. These white squares represent the data packets, which are the original data to be transmitted from the source device, after being segmented. At the beginning of the logic flow, the main control thread on the CPU responds to the transmission request. According to the pre-configured slicing strategy, it segments the original data to be transmitted, calculates control information such as the target address, creates data packets, and adds each data packet to the FPGA data packet queue (i.e., the first task queue) in sequence.

[0133] Meanwhile, the first worker thread (e.g., thread A) is responsible for data transmission and interaction with the FPGA. While the main control thread creates data packets, it retrieves data packets from the FPGA data packet queue, copies the data from the FPGA to shared memory based on the packet information, and adds the data packets to the GPU data packet queue (i.e., the second task queue). Specifically, thread A receives these white squares (i.e., the first data packets) via the bus and moves them to the shared memory inside the CPU. At this point, the data is in a pending or raw state, not yet adapted to the target device's storage format or write timing.

[0134] Understandably, when the white square (the original first data packet) enters the shared memory space, the CPU doesn't mechanically forward it. Instead, it logically rearranges or encapsulates the data based on the address characteristics and write requirements of the target device (GPU). This includes generating a metadata header containing the target address or placing the data into an ordered task queue. Through this process, the originally discrete or raw white square is transformed into a black square (the second data packet) carrying explicit control information and ready to be written. This step demonstrates the crucial role of the master control device in shielding the hardware differences between heterogeneous devices through software definition.

[0135] Next, please pay attention to the downward arrow pointing from the CPU to the GPU on the right side of the diagram, and the corresponding black squares. These black squares represent target data that is ready after being relayed and queued by the CPU. Moving to the second half of the logic flow, the second worker thread (such as thread B) retrieves these black squares from the CPU's shared memory space (or the second task queue). Thread B is responsible for data transmission and interaction with the GPU. While thread A handles data transmission between the FPGA and the CPU, it retrieves data packets from the GPU's data packet queue, copies the data from shared memory to the GPU based on the packet information, and destroys the data packets. It should be understood that because the data has already undergone rearrangement (i.e., the accurate target address has been obtained), the second worker thread can efficiently write these black squares into the GPU's video memory via the bus for subsequent high-performance computing by the GPU.

[0136] This invention utilizes the CPU as a relay station and scheduler, rearranging and managing the data output from the FPGA (source device) in memory to transform it into data suitable for the GPU (target device) to receive. This architecture avoids establishing complex hardware connections directly between the FPGA and GPU, achieving decoupling and pipelined parallelism of data interaction between heterogeneous devices, and ensuring efficient and orderly data flow within complex computing architectures.

[0137] Based on any of the above embodiments, the method further includes:

[0138] Get the current system load status;

[0139] Based on the system load status, adjust the data size of a single data packet in the preset slicing strategy, as well as the queue depth of the first and second task queues.

[0140] It should be noted that slice size and queue depth are typically set to fixed values ​​(static configuration). However, in real-world industrial applications (such as scenarios with significant load fluctuations in 6G communication), the system environment is dynamically changing. To further improve the system's adaptability and robustness, this embodiment of the invention proposes an adaptive adjustment mechanism based on load feedback.

[0141] Specifically, the main control device runs a background monitoring service or has a monitoring module integrated into the main scheduling logic. This module collects various key performance indicators of the system in real time or periodically (e.g., every 100ms) to generate the system load status.

[0142] The system load status here is a comprehensive indicator, which may include, but is not limited to, CPU utilization, memory usage, bus bandwidth utilization, and queue backlog. Among them, CPU utilization reflects the busyness of the host device in processing thread scheduling; memory usage reflects the remaining capacity of shared storage space; bus bandwidth utilization reflects the congestion of communication buses such as PCIe; and queue backlog refers to the number of tasks currently backed up in the first and second task queues.

[0143] The main control device dynamically adjusts the system's operating parameters based on the collected load status to achieve the current optimal balance. Specific adjustment logic may include: if excessive CPU load is detected (e.g., exceeding 90%), it indicates that frequent task scheduling and context switching are consuming too much computing power. In this case, the system can increase the size of individual data packets (e.g., from 4MB to 8MB). This reduces the total number of data packets, thereby lowering interrupt frequency and scheduling overhead. Although it may slightly increase the transmission latency of individual packets, it improves overall throughput.

[0144] If the system detects that the queue is idle and the service is latency-sensitive, it can reduce the packet size (e.g., from 4MB to 1MB) to achieve finer-grained pipeline processing and speed up the arrival of the first packet.

[0145] If the system detects that the data generation speed of the source device is much faster than the processing speed of the target device (i.e., data backlog has occurred), resulting in memory shortage, the system can appropriately limit (reduce) the queue depth of the first task queue, thereby putting back pressure on the source device, forcing the source device to temporarily suspend sending, and preventing the main control device from overflowing its memory.

[0146] If significant fluctuations in bus bandwidth are detected, the system can increase the queue depth and use a larger buffer to smooth out jitter, ensuring that worker threads still have data to process when the bus is busy, thus preventing pipeline interruptions.

[0147] This invention introduces an adaptive dynamic optimization mechanism. By monitoring system load in real time and dynamically adjusting packet granularity and queue depth, it can find a dynamic balance between high throughput and low latency, and protect the system through flow control mechanisms when resources are scarce. This makes the data interaction method not just a static process, but an intelligent system with self-regulating capabilities, capable of adapting to complex and ever-changing heterogeneous computing environments.

[0148] The data interaction device provided by the present invention is described below. The data interaction device described below and the data interaction method described above can be referred to in correspondence.

[0149] Based on any of the above embodiments Figure 4 This is a schematic diagram of the data interaction device provided by the present invention, as shown below. Figure 4 As shown, this device is applied to a master control device, which has a shared storage space and is communicatively connected to both a source device and a target device. The source device and the target device are heterogeneous devices. The device includes:

[0150] The request receiving unit 410 is configured to receive a transmission request for sending data from the source device to the target device.

[0151] The first copy unit 420 is used to respond to the transmission request by reading the data to be transmitted from the source device through a first working thread and writing the data to be transmitted into the shared storage space.

[0152] The second copy unit 430 is used to read the data to be transmitted from the shared storage space through the second working thread, and write the data to be transmitted into the target device;

[0153] The first worker thread and the second worker thread are executed in parallel in a pipeline manner.

[0154] The apparatus provided in this invention constructs a data transfer mechanism based on an intermediate buffer by setting up a shared storage space on the main control device and using a first working thread and a second working thread on the main control device to perform copy operations from the source device to the shared storage and from the shared storage to the target device, respectively. This mechanism first achieves logical decoupling between the source and target devices through the shared storage space, enabling data interaction between heterogeneous devices without establishing direct hardware connections or following specific point-to-point transmission protocols, thus overcoming compatibility challenges caused by differences in underlying hardware. Furthermore, by controlling the first and second working threads to execute in parallel in a pipelined manner, the system can simultaneously write the first part of data to the target device and read the next part of data from the source device. This parallel processing mode effectively masks the input / output latency of a single device and fully utilizes the bandwidth resources of the system bus, thereby significantly improving the throughput and overall efficiency of data transmission between heterogeneous devices while ensuring transmission stability.

[0155] Based on any of the above embodiments, the device further includes a data segmentation unit, the data segmentation unit being used for:

[0156] According to the transmission request, the data to be transmitted is divided into multiple data packets, so that the first working thread and the second working thread can perform data reading and writing operations at the granularity of the data packets.

[0157] Based on any of the above embodiments, the data segmentation unit is specifically used for:

[0158] The transmission request is parsed to obtain the total data volume and target address of the data to be transmitted;

[0159] The data size of a single data packet is determined based on a preset slicing strategy, and the data to be transmitted is divided into multiple data segments according to the total data volume and the data size.

[0160] Generate a data packet containing a metadata header for each data segment. The metadata header includes the packet sequence number of the data packet and the destination address.

[0161] Based on any of the above embodiments, the first copying unit is specifically used for:

[0162] The segmented data packets are added sequentially to the first task queue;

[0163] The first working thread extracts the first data packet currently to be processed from the first task queue.

[0164] Based on the information in the first data packet, the target data corresponding to the first data packet is read from the source device and written into the shared storage space.

[0165] Based on any of the above embodiments, the first copying unit is specifically used for:

[0166] While the data to be transmitted is divided into multiple data packets, the first task queue is monitored by the first working thread;

[0167] If a data packet is detected in the first task queue, the first data packet is extracted from the first task queue according to the first-in-first-out principle.

[0168] Based on any of the above embodiments, the first copying unit is further configured to:

[0169] Generate a second data packet carrying the storage address of the target data in the shared storage space, and add the second data packet to the second task queue;

[0170] The second copy unit is specifically used for:

[0171] The second data packet is extracted from the second task queue using the second worker thread;

[0172] Based on the storage address carried in the second data packet, the target data corresponding to the storage address is read from the shared storage space and written to the target device.

[0173] Based on any of the above embodiments, the second copying unit is further configured to:

[0174] The second working thread destroys the second data packet in the second task queue and releases the memory resources occupied by the target data in the shared storage space.

[0175] Based on any of the above embodiments, the device further includes a parameter adjustment unit, the parameter adjustment unit being used for:

[0176] Get the current system load status;

[0177] Based on the system load status, adjust the data size of a single data packet in the preset slicing strategy, as well as the queue depth of the first and second task queues.

[0178] Based on any of the above embodiments, this invention provides a data interaction platform, which includes a master control device, at least one source device, and at least one target device. The master control device is provided with a shared storage space, and the master control device is communicatively connected to the at least one source device and the at least one target device respectively. The master control device is used to implement the data interaction method as described in any of the above embodiments.

[0179] Specifically, the main control device is the control center of the entire platform, and it is equipped with a central processing unit (CPU) and system memory. A portion of the system memory is specifically allocated as shared storage space for temporary data transfer. The main control device is responsible for running the packet sub-logic logic, task queue management, and scheduling of the first and second worker threads in the aforementioned embodiments.

[0180] This data interaction platform can include one or more source devices. In heterogeneous computing scenarios, source devices are typically hardware accelerator cards with specific functions, such as FPGA accelerator cards (for wireless signal processing), network interface cards, or sensor acquisition cards. The source devices are responsible for generating raw data and responding to read requests from the master control device. It should be understood that the platform supports at least one source device, meaning the master control device can manage data access from multiple source devices simultaneously, such as acquiring multiple signals at the same time.

[0181] This data interaction platform may also include one or more target devices. Target devices are typically another type of heterogeneous hardware, such as GPU accelerator cards (used for AI inference or graphics rendering) or AI-specific chips. Target devices receive data from the host device and perform subsequent business processing. Similarly, the platform supports at least one target device and supports distributing data to multiple computing nodes.

[0182] The master control device is communicatively connected to at least one source device and at least one target device. In a preferred embodiment, this connection can be implemented via a PCIe bus, as PCIe offers high bandwidth and low latency. Under this hardware architecture, the master control device can be used to implement the data interaction method described in any of the above embodiments. That is, the master control device connects physically incompatible source and target devices, which may not be able to communicate directly, by running specific software drivers and middleware, using the CPU and memory as intermediaries.

[0183] The data interaction platform provided in this invention constructs a universal heterogeneous computing power interconnection architecture. This platform breaks the traditional limitation that heterogeneous devices must rely on specific hardware support, utilizing a general-purpose master control device as a central hub to achieve flexible interconnection between any source device and any target device. Simultaneously, it supports one-to-many, many-to-one, or many-to-many device expansion capabilities, giving the platform extremely strong scalability, making it particularly suitable for complex application scenarios such as 6G AI-RAN that require the integration of multiple heterogeneous computing power resources.

[0184] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a data interaction method. This method is applied to a master control device, which has a shared storage space and is communicatively connected to a source device and a target device, respectively. The source device and the target device are heterogeneous devices. The method includes: receiving a transmission request to send data from the source device to the target device; responding to the transmission request, reading the data to be transmitted from the source device through a first working thread and writing the data to be transmitted into the shared storage space; and reading the data to be transmitted from the shared storage space through a second working thread and writing the data to be transmitted into the target device; wherein the first working thread and the second working thread are executed in parallel in a pipelined manner.

[0185] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data interaction method provided by the above methods. This method is applied to a master control device, which is provided with a shared storage space and is communicatively connected to a source device and a target device, respectively. The source device and the target device are heterogeneous devices. The method includes: receiving a transmission request to send data from the source device to the target device; responding to the transmission request, reading the data to be transmitted in the source device through a first working thread and writing the data to be transmitted into the shared storage space; and reading the data to be transmitted in the shared storage space through a second working thread and writing the data to be transmitted into the target device; wherein the first working thread and the second working thread are executed in parallel in a pipelined manner.

[0187] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the data interaction method provided by the above-described methods. This method is applied to a master control device, which has a shared storage space and is communicatively connected to a source device and a target device, respectively. The source device and the target device are heterogeneous devices. The method includes: receiving a transmission request to send data from the source device to the target device; responding to the transmission request, reading the data to be transmitted from the source device through a first working thread and writing the data to be transmitted into the shared storage space; and reading the data to be transmitted from the shared storage space through a second working thread and writing the data to be transmitted into the target device; wherein the first working thread and the second working thread are executed in parallel in a pipelined manner.

[0188] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data interaction method, characterized in that, The method is applied to a master control device, which has a shared storage space and is communicatively connected to both a source device and a target device. The source device and the target device are heterogeneous devices. The method includes: Receive a data transmission request from the source device to the target device; In response to the transmission request, the first working thread reads the data to be transmitted from the source device and writes the data to be transmitted into the shared storage space; The second working thread reads the data to be transmitted from the shared storage space and writes the data to be transmitted into the target device. The first worker thread and the second worker thread are executed in parallel in a pipeline manner.

2. The data interaction method according to claim 1, characterized in that, Prior to the step of reading the data to be transmitted from the source device via the first working thread, the following steps were also included: According to the transmission request, the data to be transmitted is divided into multiple data packets, so that the first working thread and the second working thread can perform data reading and writing operations at the granularity of the data packets.

3. The data interaction method according to claim 2, characterized in that, The step of dividing the data to be transmitted into multiple data packets according to the transmission request includes: The transmission request is parsed to obtain the total data volume and target address of the data to be transmitted; The data size of a single data packet is determined based on a preset slicing strategy, and the data to be transmitted is divided into multiple data segments according to the total data volume and the data size. Generate a data packet containing a metadata header for each data segment. The metadata header includes the packet sequence number of the data packet and the destination address.

4. The data interaction method according to claim 2, characterized in that, The step of reading the data to be transmitted from the source device through the first working thread and writing the data to be transmitted into the shared storage space includes: The segmented data packets are added sequentially to the first task queue; The first working thread extracts the first data packet currently to be processed from the first task queue. Based on the information in the first data packet, the target data corresponding to the first data packet is read from the source device and written into the shared storage space.

5. The data interaction method according to claim 4, characterized in that, The step of retrieving the first data packet to be processed from the first task queue through the first worker thread includes: While the data to be transmitted is divided into multiple data packets, the first task queue is monitored by the first working thread; If a data packet is detected in the first task queue, the first data packet is extracted from the first task queue according to the first-in-first-out principle.

6. The data interaction method according to claim 4, characterized in that, After the step of writing the target data into the shared storage space, the method further includes: Generate a second data packet carrying the storage address of the target data in the shared storage space, and add the second data packet to the second task queue; The step of reading the data to be transmitted from the shared storage space and writing the data to be transmitted to the target device through the second working thread includes: The second data packet is extracted from the second task queue using the second worker thread; Based on the storage address carried in the second data packet, the target data corresponding to the storage address is read from the shared storage space and written to the target device.

7. The data interaction method according to claim 6, characterized in that, After the step of writing the target data into the target device, the method further includes: The second working thread destroys the second data packet in the second task queue and releases the memory resources occupied by the target data in the shared storage space.

8. The data interaction method according to any one of claims 1 to 7, characterized in that, Also includes: Get the current system load status; Based on the system load status, adjust the data size of a single data packet in the preset slicing strategy, as well as the queue depth of the first and second task queues.

9. A data interaction platform, characterized in that, The device includes a master control device, at least one source device, and at least one target device. The master control device is provided with a shared storage space and is communicatively connected to the at least one source device and the at least one target device. The master control device is used to implement the data interaction method as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data interaction method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data interaction method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data interaction method as described in any one of claims 1 to 8.

Images