Data processing method and device, storage medium and computer program product
By adopting the memory address sharing mechanism in RDMA data transmission, the data buffer address information is passed to avoid data copying, which solves the problem of large resource occupancy and large overhead in RDMA data transmission, and improves system performance.
Patent Information
- Application Number
- CN202411907270.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-29
AI Technical Summary
During RDMA-based data transmission, there are problems such as high resource usage and large overhead, which affect the performance of electronic devices.
By using a memory address sharing mechanism between the client and the network device, the address information of the data buffer is passed, and the data is written or read process is realized, and multiple data copies are avoided.
It reduces multiple data copies during data transmission, thereby reducing resource usage, overhead and processing delays, and improving the system performance of network equipment.
Smart Images
Figure CN119906676A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network transmission technology, and in particular to a data processing method, device, storage medium and computer program product. Background Art
[0002] Remote Direct Memory Access (RDMA) is a high-performance network transmission technology that can support efficient or high-performance data transmission between two electronic devices and reduce the direct intervention of the central processing unit (CPU). The core of RDMA is the ability of the RDMA network card, which enables electronic devices to directly access memory to achieve data transmission. This is very useful for applications that require low latency and high throughput, such as high-performance computing, large-scale data transmission, and cloud computing environments.
[0003] However, in the process of implementing data transmission between electronic devices based on RDMA, there are problems such as large resource occupation and high overhead, which will affect the performance of the electronic devices. Summary of the invention
[0004] To solve related technical problems, the embodiments of the present application provide a data processing method, device, storage medium and computer program product.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The embodiment of the present application provides a data processing method, which is applied to a first device, including:
[0007] At the computing layer, receiving a first processing request sent by a second device, the first processing request being used to request writing first data into a storage layer of the first device, the first data being stored in a first buffer, the first processing request including address information of the first buffer and feature information of the first data;
[0008] The first data is written into the storage layer by using the address information of the first buffer and the characteristic information of the first data.
[0009] In the above solution, the receiving the first processing request sent by the second device includes:
[0010] The first processing request is received through a first thread, where the first thread is at least used for information transmission between the first device and the second device.
[0011] In the above solution, the step of writing the first data into the storage layer by using the address-related information of the first buffer and the characteristic information of the first data includes:
[0012] reading the first data based on the address information of the first buffer and the characteristic information of the first data through a second thread, wherein the second thread is at least used for information transmission between the computing layer and the storage layer;
[0013] The read first data is written into the storage layer through the second thread.
[0014] In the above scheme, the method further includes:
[0015] Generate second data through a third thread, where the second data is used to describe the first buffer, and the third thread is at least used to perform business-related processing on a processing request associated with the data;
[0016] The second data is written into the storage layer through the second thread.
[0017] In the above solution, the first data is stored in the first buffer in the form of SGEs, and each SGE is used to describe the size and / or position of a data block associated with the first data.
[0018] The embodiment of the present application further provides a data processing method, which is applied to a first device and includes:
[0019] At the computing layer, receiving a second processing request sent by a second device, the second processing request being used to request reading third data from the storage layer of the first device, the second processing request including feature-related information of the third data, and the third data being stored in one or more second buffers;
[0020] Acquire address information of the one or more second buffers from the storage layer using the feature-related information of the third data;
[0021] Sending address information of the one or more second buffers to the second device.
[0022] In the above solution, the receiving the second processing request sent by the second device includes:
[0023] The second processing request is received through a first thread, where the first thread is at least used for information transmission between the first device and the second device.
[0024] In the above solution, the step of acquiring the address information of the one or more second buffers from the storage layer by using the feature-related information of the third data includes:
[0025] Sending the second processing request to the storage layer through a second thread, where the second thread is at least used for information transmission between the computing layer and the storage layer;
[0026] The address information of one or more second buffers sent by the storage layer is received through the second thread.
[0027] In the above solution, sending the second processing request to the storage layer through the second thread includes:
[0028] Performing query-related processing on the second processing request through a third thread to obtain a query result, wherein the third thread is at least used to perform business-related processing on the processing request associated with the data, and the query result represents the number and / or position of the second buffer associated with the third data;
[0029] By means of the third thread, the second processing request is segmented and processed by using the query result to obtain one or more third processing requests, each of which contains feature-related information of the third data;
[0030] The one or more third processing requests are sent to the storage layer through the second thread.
[0031] In the above scheme, the method further includes:
[0032] sorting the address information of the one or more second buffers by using the logical address information of the third data through a third thread to obtain the address information of the one or more second buffers after sorting, wherein the third thread is at least used to perform business-related processing on a processing request associated with the data;
[0033] The address information of the one or more second buffers after sorting is sent to the second device through the first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0034] In the above solution, the third data is stored in the one or more second buffers in the form of SGE, and each SGE is used to describe the size and / or position of a data block associated with the third data.
[0035] The present application also provides a device, comprising: a processor and a memory for storing a computer program that can be run on the processor,
[0036] Wherein, the processor is used to execute the steps of any of the above-mentioned data processing methods when running the computer program.
[0037] An embodiment of the present application further provides a storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above-mentioned data processing methods when executed by a processor.
[0038] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data processing methods when executed by a processor.
[0039] The data processing method, device, storage medium and computer program product provided by the embodiments of the present application are as follows: at the computing layer, a first device receives a first processing request sent by a second device, the first processing request is used to request that first data be written into the storage layer of the first device, the first data is stored in a first buffer, the first processing request includes address information of the first buffer and feature information of the first data; the first data is written into the storage layer using the address information of the first buffer and the feature information of the first data. At the computing layer, a first device receives a second processing request sent by a second device, the second processing request is used to request that third data be read from the storage layer of the first device, the second processing request includes feature-related information of the third data, the third data is stored in one or more second buffers; the address information of the one or more second buffers is obtained from the storage layer using the feature-related information of the third data; the address information of the one or more second buffers is sent to the second device. The technical solution provided by the present application adopts a memory address sharing mechanism during data transmission between a client (i.e., the second device) and a network device (i.e., the first device), and implements data writing or reading processing by transmitting address information of a data buffer to avoid data copying operations; that is, the use of zero-copy technology can avoid multiple data copies during data transmission, thereby greatly reducing resource (such as memory or bandwidth) occupancy, overhead, and processing delay, and improving the system performance of the network device. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a flow chart of the first data processing method of the embodiment of the present application;
[0041] Figure 2 This is a structural diagram of writing data in an embodiment of the present application;
[0042] Figure 3 This is a flow chart of the second data processing method according to an embodiment of the present application;
[0043] Figure 4 This is a structural diagram of reading data in an embodiment of the present application;
[0044] Figure 5 This is a schematic diagram of the structure of the first data processing device according to an embodiment of the present application;
[0045] Figure 6 This is a schematic diagram of the structure of a second data processing device according to an embodiment of the present application;
[0046] Figure 7 This is a schematic diagram of the device structure of an embodiment of the present application. DETAILED DESCRIPTION
[0047] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0048] In the related art, when using a traditional network card, the data transmission process between the client and the network device includes the following steps:
[0049] Step 1: The application puts the data to be sent into the application buffer (also known as user space), and the application is deployed on the client.
[0050] Step 2: The application passes the data to the kernel through a system call; in this process, the data is copied from the user space to the kernel space, which is the first data copy.
[0051] Step 3: The kernel's Transmission Control Protocol (TCP) / Internet Protocol (IP) protocol stack performs protocol-related processing, such as adding TCP header information, determining routing, etc.
[0052] Step 4: The protocol stack passes the data to the driver of the network device. In this process, the data needs to be copied from the kernel buffer of the protocol stack to the buffer of the network device. This is the second data copy.
[0053] Step 5: Once the data reaches the network device's buffer, it usually triggers a hardware interrupt, requiring the CPU to process the data.
[0054] Step 6: The CPU starts processing the interrupt and transmits the data to the network device. During this process, the data is transferred from the buffer of the network device to the sending queue of the network device. This is the third data copy.
[0055] Step 7: The network device sends the data to the destination address.
[0056] From the above description, it can be seen that without the involvement of direct memory access (DMA), more data copy operations (also known as actions) and CPU intervention will be introduced. This may lead to higher CPU overhead, especially in high-speed network or high data throughput scenarios; DMA reduces the direct involvement of the CPU, thereby improving the efficiency and performance of data transmission.
[0057] In addition, when using an RDMA network card, for write operations, the client transmits the data that needs to be written to the network device (such as an enterprise solid-state drive (ESSD)) through a remote procedure call (RPC). These data are represented in the form of SGE (Scatter-Gather Entries) in RDMA. At the computing layer, considering the system availability and data correctness of the network device, some metadata information needs to be generated and written to the network device together with the data; specifically, the network device requests a continuous memory buffer from the operating system through a memory allocation function at the computing layer, and copies the metadata information and the data in the SGE buffer to the memory buffer of the operating system. Then, the memory buffer of the operating system is copied to the SGE of RDMA so that it can be transmitted to the storage layer for disk writing. In the above process, two data copy operations are involved.
[0058] For read operations, since the written data may be distributed in different instances of the storage layer, when the client needs to read a continuous piece of data, the read request will be split into multiple sub-requests by the computing layer of the network device, and these sub-requests will be sent to the storage layer to obtain the relevant data buffers. Then, the computing layer will integrate the acquired data buffers into a continuous data buffer and return the integrated data buffer to the client. Specifically, after the computing layer obtains all discontinuous data buffers, it will request a certain number of discontinuous memory buffers from the operating system; then, the computing layer will copy the multiple discontinuous data buffers returned from the storage layer to these memory buffers; then, the computing layer will request a continuous memory buffer from the operating system and copy these discontinuous data buffers to the continuous memory buffer in sequence; finally, the computing layer will copy this continuous memory buffer to the SGE buffer used for RDMA and return the SGE buffer to the client. In the above process, three data copy operations are required.
[0059] From the above description, it can be seen that there are multiple data copy operations in the process of processing write operations and read operations, which will lead to the following problems:
[0060] 1. High CPU overhead: Data copying requires direct CPU intervention, that is, each time data is copied from one memory location to another, the CPU is required to perform data transfer operations. Therefore, the CPU consumes time and computing resources to perform these copy operations, which may lead to a decrease in processor performance; especially under high load conditions, high CPU overhead may become a bottleneck for the operating system, limiting overall performance.
[0061] 2. High resource consumption: Because data needs to be transferred between different memory locations, it will occupy memory bandwidth resources, causing the memory subsystem to face additional pressure, especially in high-throughput application scenarios, which will consume a lot of memory bandwidth resources. This consumption of memory bandwidth may limit other tasks' access to memory resources, further affecting system performance. In addition, data copying also requires the operating system to participate in memory allocation and management, which involves operations such as requesting memory buffers from the operating system, releasing memory, and managing memory pages; the above process takes time and computing resources, and may cause memory fragmentation problems, making memory use inefficient.
[0062] 3. Data redundancy: Multiple data copies (also called multiple data copies) mean that the same data may be stored multiple times in the memory. Each data copy will result in redundant storage of the data, which not only wastes memory space, but may also cause data inconsistency problems. That is, if the data of one copy is modified, other copies may not be updated synchronously in time.
[0063] 4. Long processing delay: Because data must be copied from one memory location to another, it is easy to introduce a certain amount of delay. For applications that require low-latency responses, this additional delay may become a performance bottleneck, thereby reducing the real-time performance of the system.
[0064] 5. High complexity of memory management: Since different memory buffers need to be tracked during the data copy process to ensure the integrity and correctness of the data, the complexity of memory management will be increased, making the memory difficult to maintain and debug.
[0065] In summary, in the RDMA-based distributed storage architecture, multiple data copies introduce multiple problems in the data transmission process. These problems need to be carefully considered when designing and optimizing the system to improve system performance, efficiency, and maintainability.
[0066] Based on this, in various embodiments of the present application, during the data transmission between the client and the network device, the data buffer is located by address information of the data buffer to implement data writing and reading processing, without the need to copy the data in the data buffer multiple times, so that the number of data copies during the writing and reading operations can be minimized, thereby reducing overhead and resource consumption. In this way, even under high load conditions, the network device can efficiently process data requests and provide better system performance and response time.
[0067] The present application embodiment provides a data processing method, such as Figure 1 As shown, applied to a first device, the method includes:
[0068] Step 101: at a computing layer, receiving a first processing request sent by a second device, wherein the first processing request is used to request writing first data into a storage layer of the first device, the first data being stored in a first buffer, and the first processing request including address information of the first buffer and feature information of the first data;
[0069] Step 102: Write the first data into the storage layer by using the address information of the first buffer and the characteristic information of the first data.
[0070] Among them, in actual application, the first device can be called a network device, a hardware device or an electronic device, etc. The first device can include a disk or a hard disk, etc. The embodiment of the present application does not limit the type of the first device, as long as its function is realized. In addition, the first device can support distributed storage, that is, the storage layer of the first device can include multiple instances (also called storage services or storage engines), can interact with multiple instances contained in the computing layer (also called computing services or computing engines) and store data in multiple buffers to realize distributed storage of data.
[0071] In actual application, the second device can be called a client, a user device, etc. The embodiment of the present application does not limit the name of the second device as long as its function is realized.
[0072] In actual application, in step 101, at the computing layer, the first device may receive the first processing request through a thread used to communicate with the second device; wherein the first device may communicate with the second device based on RDMA.
[0073] Specifically, in one embodiment, the receiving a first processing request sent by the second device includes:
[0074] The first processing request is received through a first thread, where the first thread is at least used for information transmission between the first device and the second device.
[0075] Among them, the first thread can be understood as the thread of the computing layer, which can realize the communication between the computing layer and the second device. The first thread can be called a request acceptance thread. The number of the first threads can be one or more, which is not limited in this embodiment of the present application.
[0076] In actual application, when the first data needs to be written, the second device may apply for the first buffer for the first data, and store the first data in the first buffer in a specific data format.
[0077] In one embodiment, the first data is stored in the first buffer in the form of SGEs, and each SGE is used to describe the size and / or position of a data block associated with the first data.
[0078] In actual application, when the first buffer stores data in the form of SGE, the first buffer can be called a first SGE buffer, a first SGE data buffer, etc.; wherein SGE can be understood as a separate element (also known as an item) in SGL, which can describe the size (also known as length) and / or position of a data block associated with the first data, such as the starting address of the data block in the buffer (also known as a memory address) or the length of the data block.
[0079] It should be noted that SGL is a data structure, usually composed of multiple SGEs, which can describe the location and size of multiple data blocks that the device needs to transmit, so that the device can know how to read data and how to transmit data; among them, SGL at least supports the device to read data from multiple locations in a scattered (English can be expressed as Scatter) manner, and write data to multiple locations in an aggregated (English can be expressed as Gather) manner. Therefore, SGL and SGE can be understood as key elements of RDMA-based data operations, which can support devices to transmit data in an efficient manner and support reading and writing data from multiple locations. This ability to disperse and aggregate is very important for processing large-scale data transmission and high-performance computing. By using SGL and SGE, devices can manage and transmit data in a very flexible way, while reducing CPU intervention and data copying.
[0080] In actual application, after storing the first data, the second device can send the first processing request to the first device, so that the first device can receive the first processing request through the first thread; wherein, the characteristic information of the first data can include size information (which can be expressed as size in English) and offset information (which can be expressed as offset in English), etc. of the first data, and the characteristic information of the first data and the address information of the first buffer can be sent in the form of SGE; that is, the first processing request can include multiple SGEs, and each SGE can include the address information of the first buffer corresponding to a data block associated with the first data and the characteristic information corresponding to the data block, so that the transmission of multiple discontinuous data blocks can be supported.
[0081] In actual application, in step 102, at the computing layer, the first device may write the first data through a thread used for information transmission between the computing layer and the storage layer.
[0082] Specifically, in one embodiment, using the address-related information of the first buffer and the characteristic information of the first data to write the first data into the storage layer includes:
[0083] reading the first data based on the address information of the first buffer and the characteristic information of the first data through a second thread, wherein the second thread is at least used for information transmission between the computing layer and the storage layer;
[0084] The read first data is written into the storage layer through the second thread.
[0085] Among them, the second thread can be understood as the thread of the computing layer, which can realize the read and write interaction between the computing layer and the storage layer. The second thread can be called an interactive thread. The number of the second threads can be one or more, which is not limited in the embodiment of the present application.
[0086] In actual application, in order to write the first data, the first device can reference the address information of the first buffer from the first processing request through the third thread, and pass the address information of the first buffer to the second thread, so that the second thread locates the physical address of the first buffer based on the address information of the first buffer, and then reads the first data in combination with the characteristic information of the first data and realizes the writing of the data; wherein, the third thread can be understood as the thread of the computing layer, the number of the third threads can be one or more, and can perform business-related processing (such as verification or query processing) on the processing request; the third thread can be called a task processing thread, which can constitute a three-layer thread model of the computing layer with the first thread and the second thread, so that the data writing process and the reading process can be executed in order based on the three-layer thread model; that is, using different threads to perform different functions can ensure the correctness of data processing, prevent data confusion or loss, help maintain data integrity and consistency, and improve the maintainability and scalability of the system.
[0087] In actual application, during the process of writing the first data, the first device may also generate metadata associated with the first data to ensure the correctness of the data.
[0088] Based on this, in one embodiment, the method may further include:
[0089] Generate second data through a third thread, where the second data is used to describe the first buffer, and the third thread is at least used to perform business-related processing on a processing request associated with the data;
[0090] The second data is written into the storage layer through the second thread.
[0091] The second data may be understood as metadata, and the second data may at least indicate a starting address or an ending address of the first data in the first buffer.
[0092] Here, through the third thread, the first device can construct the second data, treat the first data and the second data as a whole data, and write the whole data into the first buffer through the second thread.
[0093] In actual application, after completing writing of the first data and the second data, the first device may send a first request response to the second device through the first thread, so that the second device can know that the first data is successfully written.
[0094] For example, Figure 2 As shown, under the architecture of the three-layer thread model, after the first device (also referred to as a block server) receives the multiple SGEs included in the first processing request through the first thread (which can be described as rev_worker), the address information of the first buffer in the SGE is directly referenced (which can be described as ref) through the third thread (which can be described as rpc_worker), and the address information of the first buffer is passed to the second thread (which can be described as task_worker), and the second thread locates the physical address of the first buffer and forwards the first data to the storage engine in the storage layer to write the first data.
[0095] In addition, through the third thread, the first device can also record the mapping relationship between the physical address information of the first buffer and the logical address information of the first data to facilitate subsequent data query.
[0096] In the embodiment of the present application, the data writing scheme constructed based on the RDMA memory sharing technology can avoid the operation of copying data from the first buffer to the memory, thereby minimizing the number of times the data is copied between the computing layer and the storage layer. In this way, the redundant storage of data in the memory can be reduced, the CPU overhead can be reduced, and at the same time, the consumption of memory bandwidth can be reduced, which not only improves the efficiency of the write operation, but also helps to reduce the latency of the system, thereby bringing significant performance and efficiency advantages to the system.
[0097] Accordingly, the embodiment of the present application also provides a data processing method, which is applied to a first device, such as Figure 3 As shown, the following steps are included:
[0098] Step 301: at the computing layer, receiving a second processing request sent by a second device, the second processing request being used to request reading third data from the storage layer of the first device, the second processing request including feature-related information of the third data, the third data being stored in one or more second buffers;
[0099] Step 302: Acquire address information of the one or more second buffers from the storage layer using the feature-related information of the third data;
[0100] Step 303: Send address information of the one or more second buffers to the second device.
[0101] In actual application, in the reading scenario of the computing layer, the second processing request can be obtained through the first thread.
[0102] Specifically, in one embodiment, the receiving the second processing request sent by the second device includes:
[0103] The second processing request is received through a first thread, where the first thread is at least used for information transmission between the first device and the second device.
[0104] In actual application, the third data may be stored in multiple second buffers in a dispersed manner, or may be stored in one second buffer in an aggregated manner; wherein the third data may be stored in the one or more second buffers in a specific data format.
[0105] Specifically, in one embodiment, the third data may be stored in the one or more second buffers in the form of SGEs, and each SGE is used to describe the size and / or position of a data block associated with the third data.
[0106] In which, when the second buffer stores data in the form of SGE, the second buffer may be called a second SGE buffer, a second SGE data buffer, etc., which is not limited in the embodiment of the present application.
[0107] In actual application, in step 302, the first device may request the data segment associated with the third data from the storage layer through the second thread.
[0108] Specifically, in one embodiment, the acquiring, from the storage layer, address information of the one or more second buffers using the feature-related information of the third data includes:
[0109] Sending the second processing request to the storage layer through a second thread, where the second thread is at least used for information transmission between the computing layer and the storage layer;
[0110] The address information of one or more second buffers sent by the storage layer is received through the second thread.
[0111] In actual application, since the third data may be distributed and stored in a plurality of second buffers, the first device may send the second processing request according to the storage status of the third data.
[0112] Specifically, in one embodiment, sending the second processing request to the storage layer through the second thread includes:
[0113] Performing query-related processing on the second processing request through a third thread to obtain a query result, wherein the third thread is at least used to perform business-related processing on the processing request associated with the data, and the query result represents the number and / or position of the second buffer associated with the third data;
[0114] By means of the third thread, the second processing request is segmented and processed by using the query result to obtain one or more third processing requests, each of which contains feature-related information of the third data;
[0115] The one or more third processing requests are sent to the storage layer through the second thread.
[0116] In actual application, the third processing request can be understood as a sub-request obtained by splitting the second processing request.
[0117] Here, through the third thread, the first device can perform verification processing on the second processing request; if the verification passes, query-related processing is performed on the second processing request to obtain the query result, thereby determining the storage distribution of the third data. Once the storage situation of the third data is determined, through the third thread, the first device can split the second processing request into one or more third processing requests, and transmit the one or more third processing requests in parallel to one or more second threads, so as to request the third data from one or more instances of the storage layer through one or more second threads; wherein the number of third processing requests is associated with the distribution of the third data, for example, if the third data is distributed and stored in 5 different second buffers, the number of third processing requests is 5.
[0118] In actual application, through the second thread, the first device can obtain the address information of the one or more second buffers from the storage layer; wherein the address information of the one or more second buffers can be transmitted in the form of SGE; that is, the first device can receive one or more SGEs returned by the storage layer, each SGE can contain the address information of the second buffer corresponding to a data block associated with the third data. In the process of transmitting the third data from the storage layer to the computing layer, no redundant copy will be generated in the memory of the computing layer.
[0119] In the embodiment of the present application, through the three-layer thread model, the read request is divided into multiple sub-requests for parallel processing, which can improve the processing efficiency of the read operation and enable the system to better cope with high-concurrency read requests. In addition, by processing multiple sub-requests in parallel, it is also possible to fully utilize multi-core CPU resources, reduce resource waste and improve system resource utilization. The above data processing scheme has potential application prospects in many fields (especially those requiring high performance and low latency), including cloud computing, big data processing, high-performance storage, scientific computing and other fields.
[0120] In practical applications, since the actual storage order of the third data in the second buffer may be different from the logical order of the third data, the first device may adjust the order of the second buffer to ensure the orderliness of the data.
[0121] Based on this, in one embodiment, the method may further include:
[0122] sorting the address information of the one or more second buffers by using the logical address information of the third data through a third thread to obtain the address information of the one or more second buffers after sorting, wherein the third thread is at least used to perform business-related processing on a processing request associated with the data;
[0123] The address information of the one or more second buffers after sorting is sent to the second device through the first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0124] In actual application, using the second processing request, the first device can determine the logical address information of the third data through the third thread, and the logical address information includes a logical block address (LBA); then, using the logical address information of the third data, the first device can sort the address information of the one or more second buffers in order from small to large; then, through the first thread, the address information of the one or more second buffers after sorting is fed back to the second device. In the above process, through the zero-copy method, data can be efficiently organized, transmitted and returned, avoiding multiple copies of data in the system memory, reducing CPU overhead and memory bandwidth consumption, and thus significantly improving the efficiency and performance of the read operation.
[0125] For example, Figure 4 As shown, under the architecture of the three-layer thread model, for the second processing request, through the second thread, the first device can obtain multiple SGE buffers (which can be understood as data fragments of the third data returned in the form of SGE) from the storage layer; through the third thread, the address information of the SGE buffer is sorted according to the size of the LBA address to generate an SGE list; through the first thread, the SGE list is added to the SGL, and the SGL is sent to the second device.
[0126] The data processing method provided by the embodiment of the present application is as follows: at the computing layer, a first device receives a first processing request sent by a second device, the first processing request is used to request that first data be written into the storage layer of the first device, the first data is stored in a first buffer, the first processing request includes address information of the first buffer and feature information of the first data; the first data is written into the storage layer using the address information of the first buffer and the feature information of the first data. At the computing layer, a first device receives a second processing request sent by a second device, the second processing request is used to request that third data be read from the storage layer of the first device, the second processing request includes feature-related information of the third data, the third data is stored in one or more second buffers; the address information of the one or more second buffers is obtained from the storage layer using the feature-related information of the third data; the address information of the one or more second buffers is sent to the second device. The technical solution provided by the present application adopts a memory address sharing mechanism during data transmission between a client (i.e., the second device) and a network device (i.e., the first device), and implements data writing or reading processing by transmitting address information of a data buffer to avoid data copying operations; that is, the use of zero-copy technology can avoid multiple data copies during data transmission, thereby greatly reducing resource (such as memory or bandwidth) occupancy, overhead, and processing delay, and improving the system performance of the network device.
[0127] In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides a data processing device, which is arranged on a first device, such as Figure 5 As shown, the device comprises:
[0128] A first receiving unit 501 is configured to receive, at a computing layer, a first processing request sent by a second device, wherein the first processing request is used to request writing first data into a storage layer of the first device, the first data being stored in a first buffer, and the first processing request including address information of the first buffer and feature information of the first data;
[0129] The writing unit 502 is used to write the first data into the storage layer by using the address information of the first buffer and the characteristic information of the first data.
[0130] In one embodiment, the first receiving unit 501 is used to receive the first processing request through a first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0131] In one embodiment, the writing unit 502 is used to:
[0132] reading the first data based on the address information of the first buffer and the characteristic information of the first data through a second thread, wherein the second thread is at least used for information transmission between the computing layer and the storage layer;
[0133] The read first data is written into the storage layer through the second thread.
[0134] In one embodiment, the writing unit 502 is further used for:
[0135] Generate second data through a third thread, where the second data is used to describe the first buffer, and the third thread is at least used to perform business-related processing on a processing request associated with the data;
[0136] The second data is written into the storage layer through the second thread.
[0137] In actual application, the first receiving unit 501 can be implemented by a communication interface in a data processing device; and the writing unit 502 can be implemented by a processor in the data processing device.
[0138] In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides a data processing device, which is arranged on a first device, such as Figure 6 As shown, the device comprises:
[0139] The second receiving unit 601 is configured to receive, at the computing layer, a second processing request sent by a second device, where the second processing request is used to request reading third data from the storage layer of the first device, where the second processing request includes feature-related information of the third data, and the third data is stored in one or more second buffers;
[0140] An acquiring unit 602 is configured to acquire address information of the one or more second buffers from the storage layer by using feature-related information of the third data;
[0141] The sending unit 603 is configured to send address information of the one or more second buffers to the second device.
[0142] In one embodiment, the second receiving unit 601 is used to receive the second processing request through a first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0143] In one embodiment, the obtaining unit 602 is used to:
[0144] Sending the second processing request to the storage layer through a second thread, where the second thread is at least used for information transmission between the computing layer and the storage layer;
[0145] The address information of one or more second buffers sent by the storage layer is received through the second thread.
[0146] In one embodiment, the acquisition unit 602 is used to perform query-related processing on the second processing request through a third thread to obtain a query result, the third thread is at least used to perform business-related processing on the processing request associated with the data, and the query result represents the number and / or position of the second buffer associated with the third data; through the third thread, the second processing request is segmented using the query result to obtain one or more third processing requests, each third processing request contains feature-related information of the third data;
[0147] The sending unit 603 is used to send the one or more third processing requests to the storage layer through the second thread.
[0148] In one embodiment, the acquisition unit 602 is further used to sort the address information of the one or more second buffers by using the logical address information of the third data through a third thread to obtain the address information of the one or more second buffers after sorting, and the third thread is at least used to perform business-related processing on the processing request associated with the data;
[0149] The sending unit 603 is further used to send address information of one or more second buffers after sorting to the second device through a first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0150] In actual application, the second receiving unit 601 and the sending unit 603 can be implemented by a communication interface in a data processing device; the acquiring unit 602 can be implemented by a processor in the data processing device in combination with a communication interface.
[0151] It should be noted that: when the data processing device provided in the above embodiment performs data processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0152] Based on the hardware implementation of the above program module, and in order to implement the method on the first device side of the embodiment of the present application, the embodiment of the present application also provides a device (ie, the first device mentioned above), such as Figure 7 As shown, the first device 700 includes:
[0153] A first communication interface 701, capable of interacting with a second device;
[0154] A first processor 702 is connected to the first communication interface 701 to implement interaction with the second device, and is used to execute the method provided by one or more technical solutions on the first device side when running a computer program;
[0155] A first memory 703 , on which the computer program is stored.
[0156] Specifically, in the data writing scheme, the first communication interface 701 is used to receive, at the computing layer, a first processing request sent by the second device, the first processing request being used to request writing first data into the storage layer of the first device, the first data being stored in a first buffer, and the first processing request including address information of the first buffer and feature information of the first data;
[0157] The first processor 702 is configured to write the first data into the storage layer by using the address information of the first buffer and the characteristic information of the first data.
[0158] In one embodiment, the first communication interface 701 is used to receive the first processing request through a first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0159] In one embodiment, the first processor 702 is configured to:
[0160] reading the first data based on the address information of the first buffer and the characteristic information of the first data through a second thread, wherein the second thread is at least used for information transmission between the computing layer and the storage layer;
[0161] The read first data is written into the storage layer through the second thread.
[0162] In one embodiment, the first processor 702 is further configured to:
[0163] Generate second data through a third thread, where the second data is used to describe the first buffer, and the third thread is at least used to perform business-related processing on a processing request associated with the data;
[0164] The second data is written into the storage layer through the second thread.
[0165] In the data reading scheme, the first communication interface 701 is used to receive, at the computing layer, a second processing request sent by a second device, the second processing request being used to request reading third data from the storage layer of the first device, the second processing request including feature-related information of the third data, and the third data being stored in one or more second buffers;
[0166] The first processor 702 is used to obtain the address information of the one or more second buffers from the storage layer by using the feature-related information of the third data; and send the address information of the one or more second buffers to the second device through the first communication interface 701.
[0167] In one embodiment, the first communication interface 701 is used to receive the second processing request through a first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0168] In one embodiment, the first communication interface 701 is used to:
[0169] Sending the second processing request to the storage layer through a second thread, where the second thread is at least used for information transmission between the computing layer and the storage layer;
[0170] The address information of one or more second buffers sent by the storage layer is received through the second thread.
[0171] In one embodiment, the first processor 702 is used to perform query-related processing on the second processing request through a third thread to obtain a query result, the third thread is at least used to perform business-related processing on the processing request associated with the data, and the query result represents the number and / or position of the second buffer associated with the third data; through the third thread, the second processing request is segmented using the query result to obtain one or more third processing requests, each third processing request containing feature-related information of the third data;
[0172] The first communication interface 701 is used to send the one or more third processing requests to the storage layer through the second thread.
[0173] In one embodiment, the first processor 702 is further used to sort the address information of the one or more second buffers by using the logical address information of the third data through a third thread to obtain the address information of the one or more second buffers after sorting, and the third thread is at least used to perform business-related processing on the processing request associated with the data;
[0174] The first communication interface 701 is further used to send address information of one or more second buffers after sorting to the second device through a first thread, and the first thread is at least used for information transmission between the first device and the second device.
[0175] It should be noted that the specific processing process of the processor 702 and the communication interface 701 can be understood by referring to the above method.
[0176] Of course, in actual application, the various components in the device 700 are coupled together through the bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .
[0177] The memory 703 in the embodiment of the present application is used to store various types of data to support the operation of the device 700. Examples of such data include: any computer program used to operate on the device 700.
[0178] The method disclosed in the above embodiment of the present application can be applied to the processor 702, or implemented by the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 702 or an instruction in the form of software. The above-mentioned processor 702 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 703, and the processor 702 reads the information in the memory 703 and completes the steps of the above method in combination with its hardware.
[0179] In an exemplary embodiment, the device 700 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.
[0180] It can be understood that the memory (memory 703) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.
[0181] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 703 storing a computer program, and the computer program can be executed by a processor 702 of a device 700 to complete the steps of the first device-side method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.
[0182] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 702 of the device 700 to complete the steps described in the aforementioned first device-side method.
[0183] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0184] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.
[0185] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.
Claims
1. A data processing method, characterized in that: Applied to a first device, comprising: At the computing layer, receiving a first processing request sent by a second device, the first processing request being used to request writing first data into a storage layer of the first device, the first data being stored in a first buffer, the first processing request including address information of the first buffer and feature information of the first data; The first data is written into the storage layer by using the address information of the first buffer and the characteristic information of the first data.
2. The method according to claim 1, characterized in that The receiving a first processing request sent by the second device includes: The first processing request is received through a first thread, where the first thread is at least used for information transmission between the first device and the second device.
3. The method according to claim 1, characterized in that The step of writing the first data into the storage layer by using the address-related information of the first buffer and the characteristic information of the first data includes: reading the first data based on the address information of the first buffer and the characteristic information of the first data through a second thread, wherein the second thread is at least used for information transmission between the computing layer and the storage layer; The read first data is written into the storage layer through the second thread.
4. The method according to claim 3, characterized in that The method further comprises: Generate second data through a third thread, where the second data is used to describe the first buffer, and the third thread is at least used to perform business-related processing on a processing request associated with the data; The second data is written into the storage layer through the second thread.
5. The method according to any one of claims 1 to 4, characterized in that: The first data is stored in the first buffer in the form of SGEs, and each SGE is used to describe the size and / or position of a data block associated with the first data.
6. A data processing method, characterized in that: Applied to a first device, comprising: At the computing layer, receiving a second processing request sent by a second device, the second processing request being used to request reading third data from the storage layer of the first device, the second processing request including feature-related information of the third data, and the third data being stored in one or more second buffers; Acquire address information of the one or more second buffers from the storage layer using the feature-related information of the third data; Sending address information of the one or more second buffers to the second device.
7. The method according to claim 6, characterized in that The receiving a second processing request sent by the second device includes: The second processing request is received through a first thread, where the first thread is at least used for information transmission between the first device and the second device.
8. The method according to claim 6, characterized in that The acquiring, from the storage layer, address information of the one or more second buffers by using the feature-related information of the third data includes: Sending the second processing request to the storage layer through a second thread, where the second thread is at least used for information transmission between the computing layer and the storage layer; The address information of one or more second buffers sent by the storage layer is received through the second thread.
9. The method according to claim 8, characterized in that The sending the second processing request to the storage layer through the second thread includes: Performing query-related processing on the second processing request through a third thread to obtain a query result, wherein the third thread is at least used to perform business-related processing on the processing request associated with the data, and the query result represents the number and / or position of the second buffer associated with the third data; By means of the third thread, the second processing request is segmented and processed by using the query result to obtain one or more third processing requests, each of which contains feature-related information of the third data; The one or more third processing requests are sent to the storage layer through the second thread.
10. The method according to claim 6, characterized in that The method further comprises: sorting the address information of the one or more second buffers by using the logical address information of the third data through a third thread to obtain the address information of the one or more second buffers after sorting, wherein the third thread is at least used to perform business-related processing on a processing request associated with the data; The address information of the one or more second buffers after sorting is sent to the second device through the first thread, and the first thread is at least used for information transmission between the first device and the second device.
11. The method according to any one of claims 6 to 11, characterized in that: The third data is stored in the one or more second buffers in the form of SGEs, and each SGE is used to describe the size and / or position of a data block associated with the third data.
12. A device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 5, or implements the steps of the method described in any one of claims 6 to 12.
13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 12.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 12.