Data transmission method and device and electronic equipment

By establishing an RDMA data transmission channel between the client and the server, directly transmitting data, the problems of frequent data copy times and network delay in traditional methods are solved, and efficient large-data transmission is achieved.

CN119088581APending Publication Date: 2024-12-06MGI TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310661168.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-06
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional network data transmission methods lead to a large CPU and memory burden when transmitting large amounts of data, resulting in network delays, and the number of data copies is large, which affects efficiency.

Method used

By establishing an RDMA data transmission channel between the client and the server, the target data is written to the second data buffer of the sending process, and directly transmitted to the first data buffer of the receiving process through the RDMA data transmission channel, reducing the number of data copies .

Benefits of technology

It reduces the transmission delay of large data volume, improves bandwidth utilization, and is suitable for the transmission of large data volume.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088581A_ABST
    Figure CN119088581A_ABST
Patent Text Reader

Abstract

The invention relates to a data transmission method and device and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: obtaining target data to be subjected to data transmission; determining data transmission position information of the RDMA communication connection, wherein the data transmission position information comprises a first data buffer area of a receiving process corresponding to the server, a second data buffer area of a sending process corresponding to the client, and an RDMA data transmission channel between the first data buffer area and the second data buffer area; and writing the target data into a second data buffer area of the sending process so as to transmit the target data in the second data buffer area to the first data buffer area through the RDMA data transmission channel by utilizing the sending process. According to the data transmission method and device, the RDMA data transmission channel is established between the sending process and the receiving process, the target data is written into the cache address of the second data buffer area, and the target data is directly transmitted to the first data buffer area, so that the copying times of the data in the transmission process can be reduced, and the transmission delay of a large data volume is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of network communication technology, and in particular to a data transmission method, device and electronic device. Background Art

[0002] The traditional network data transmission method is to use the Transmission Control Protocol (TCP) and the Internet Protocol (IP) to establish a connection between the data sender and the data receiver, and to move data by copying the data through the central processing unit (CPU). This data transmission method has high requirements on the speed and performance of the network. When the amount of data is large and the interaction is frequent, it will cause a heavy burden on the CPU and memory, and there will be network delays, which is not friendly to the transmission of large amounts of data. Summary of the invention

[0003] In view of this, the present disclosure provides a data transmission method, device and electronic device, the main purpose of which is to reduce the number of data copies during data transmission and reduce the transmission delay of large amounts of data.

[0004] According to a first aspect of the present disclosure, a data transmission method is provided, the method being executed by a client, the method comprising:

[0005] Acquire target data to be transmitted;

[0006] Determine data transmission location information of an RDMA communication connection established with a server, the data transmission location information including a first data buffer of a receiving process corresponding to the server, a second data buffer of a sending process corresponding to the client, and an RDMA data transmission channel between the sending process and the receiving process;

[0007] The target data is written into the second data buffer of the sending process, so that the sending process is used to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

[0008] According to a second aspect of the present disclosure, a data transmission method is provided, the method being executed by a server, the method comprising:

[0009] A receiving process is used to receive target data transmitted by a client, wherein the target data is written by the client into a second data buffer of a sending process, and then the sending process is used to transmit the target data to a first data buffer of the receiving process through an RDMA data transmission channel between the sending process and the receiving process.

[0010] According to a third aspect of the present disclosure, a data transmission device is provided, the device being executed by a client, and the device comprising:

[0011] An acquisition module, used for acquiring target data to be transmitted;

[0012] A determination module, used to determine data transmission information of an RDMA communication connection established with a server, wherein the data transmission information includes a first data buffer of a receiving process corresponding to the server, and an RDMA data transmission channel between the sending process and the receiving process;

[0013] The transmission module is used to write the target data into the second data buffer of the sending process, so as to utilize the sending process to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

[0014] According to a fourth aspect of the present disclosure, a data transmission device is provided, the device being executed by a server, and the device comprising:

[0015] The receiving module is used to use the receiving process to receive the target data transmitted by the client, wherein the target data is written by the client into the second data buffer of the sending process, and then the target data is transmitted by the sending process to the first data buffer of the receiving process through the RDMA data transmission channel between the sending process and the receiving process.

[0016] According to the fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect above, or execute the method described in the second aspect above.

[0017] According to the sixth aspect of the present disclosure, the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect above, or execute the method described in the second aspect above.

[0018] According to a seventh aspect of the present disclosure, the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect above, or executes the method described in the second aspect above.

[0019] The data transmission method, device and electronic device provided by the present disclosure include: the client obtains the target data to be transmitted, the client determines the data transmission location information of the RDMA communication connection established between the client and the server, the data transmission location information includes the first data buffer corresponding to the receiving process of the server, the second data buffer corresponding to the sending process of the client, and the RDMA data transmission channel between the sending process and the receiving process, the target data is written into the second data buffer of the sending process, so as to use the sending process to transmit the target data in the second data buffer to the first data buffer of the server through the RDMA data transmission channel. The scheme of the present disclosure, by establishing an RDMA data transmission channel between the sending process of the client and the receiving process of the server, writes the target data into the cache address of the second data buffer of the sending process, and directly transmits the target data to the first data buffer of the receiving process through the cache address, thereby reducing the number of data copies during the transmission process and reducing the transmission delay of large amounts of data.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A schematic diagram of a data transmission method provided by an embodiment of the present disclosure is shown;

[0024] Figure 2 A schematic diagram showing the overall principle structure of a data transmission method provided by an embodiment of the present disclosure is shown;

[0025] Figure 3 A schematic diagram showing a flow chart of another data transmission method provided by an embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram showing a flow chart of another data transmission method provided by an embodiment of the present disclosure;

[0027] Figure 5A schematic diagram of the structure of a data transmission device provided in an embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of the structure of another data transmission device provided by an embodiment of the present disclosure;

[0029] Figure 7 A schematic diagram of the structure of another data transmission device provided in an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of the structure of another data transmission device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The data transmission method, device and electronic device according to the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0033] The traditional network data transmission method is to use the Transmission Control Protocol (TCP) and the Internet Protocol (IP) to establish a connection between the data sender and the data receiver, and to move data by copying the data through the central processing unit (CPU). This data transmission method has high requirements on the speed and performance of the network. When the amount of data is large and the interaction is frequent, it will cause a heavy burden on the CPU and memory, and there will be network delays, which is not friendly to the transmission of large amounts of data.

[0034] In order to solve the above technical problems, the present disclosure provides a data transmission method, device and electronic device, which can reduce the number of data copies during data transmission and reduce the transmission delay of large amounts of data.

[0035] like Figure 1 , an embodiment of the present disclosure provides a data transmission method, which is executed by a client and includes:

[0036] Step 101: Obtain target data to be transmitted.

[0037] Among them, the target data may be image data information, text data information, audio data information, etc. In the following embodiments of the present disclosure, the target data is image data information as an example to illustrate the technical solution of the present disclosure, but it does not constitute a specific limitation on the technical solution in the present disclosure. For the embodiments of the present disclosure, the client (such as an optical image acquisition program) can be used to obtain image data information to be transmitted for data. In a specific application scenario, the image data may be an ATCG base image generated by a camera, which can be applied to the field of gene sequencing, or it can also be applied to other fields. In the following embodiment steps in the disclosure, the field of gene sequencing is taken as an example to illustrate the technical solution in the present disclosure, but it does not constitute a specific limitation. In the existing gene sequencing field, there are multiple sequencing platforms, such as the sequencing platform of MGI, the ROCHE 454 sequencing platform, the ILLUMINATM SOLEXATM sequencing platform, the SOLIDTM sequencing platform of LIFE TECHNOLOGIES / APPLIED BIOSYSTEMS, the SMRTTM sequencing platform of PACIFIC BIOSCIENCES, the POLLONATOR Polony sequencing platform, the COMPLETEGENOMICS sequencing platform, the sequencing platform of INTELLIGENT BIOSYSTEMS, the HELICOS sequencing platform or any other gene sequencer in the art. These sequencing platforms will generate a large number of ATCG base images (in the sequencing field, the four ATCG base images cover all base types), which is a huge burden for transmission software. For example, the existing data transmission method mainly transmits data through TCP / IP, which requires a large memory bus bandwidth. The data is copied and transmitted between the system and the processor, which has high requirements on the speed and performance of the network. For ATCG base images with large data volume, it will cause a large burden on the CPU and memory, and there will be network delay, which is very unfriendly to the transmission of large data volume. The present invention uses a high-speed RDMA network card to establish a sending program on the Windows side, a receiving program on the Linux side, and controls the sending and receiving of image data by adapting the protocol of the sequencer scheduling interface. Different from the layer-by-layer copy transmission of TCP / IP communication, data reaches the destination through semaphores, events, etc. The present invention performs signal verification on each node in the communication sequence, does not make any copy of the image data, and only brings in the cache address where the data is located, thereby reducing data copying, improving bandwidth utilization, and being suitable for the transmission of large amounts of data.

[0038] For the disclosed embodiments, a gene sequencer is an instrument for gene sequencing. In specific application scenarios, gene sequencing / nucleic acid sequencing includes DNA sequencing and RNA sequencing, and is one of the important methods for studying nucleic acids. DNA sequencing usually sets a number of reaction wells for accommodating DNA fragment molecules on a microfluidic chip. These reaction wells can usually only accommodate one DNA fragment molecule or a magnetic bead with a DNA fragment molecule attached; a solution containing the DNA fragment molecule to be sequenced flows over the surface of the above chip, and the DNA fragment molecule falls into the reaction well; various detection reagents flow through the microfluidic chip respectively, and the reagents flowing into the reaction well react chemically with the DNA fragment molecules, emitting light of various specific colors. A microscope camera is set just above the chip to record the light emitted by each detection reagent when it flows through the reaction well, and the base sequence of the DNA fragment can be detected.

[0039] Step 102: Determine the data transmission location information of the RDMA communication connection established with the server, where the data transmission location information includes a first data buffer of the server corresponding to the receiving process, a second data buffer of the client corresponding to the sending process, and an RDMA data transmission channel between the sending process and the receiving process.

[0040] The first data buffer may be a first Remote Direct Memory Access (RDMA) cache, the second data buffer may be a second RDMA cache, and the RDMA data transmission channel is a pre-set channel for data transmission.

[0041] For the embodiments of the present disclosure, Figure 2 As shown, the client and the server have established an RDMA communication connection. The client (Windows side) acts as the sender of image data, and the server (Linux side, which can be a BCS server) acts as the receiver of image data information. A high-speed RDMA network card can be used to establish a sending process on the client and a receiving process on the server. The received image data information is transmitted from the sending process of the client to the receiving process of the server through the RDMA data transmission channel.

[0042] Step 103: Write the target data into the second data buffer of the sending process, so as to use the sending process to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

[0043] The execution subject of the present disclosure may be a client, and the working process of the target data transmission is as follows: when the client executes an RDMA read or write request, no data copy is performed, and the RDMA request is sent to the RDMA network card without any kernel memory participation. The RDMA network card reads the cached image data information, the image data information at least includes the image data cache address and the image data itself, writes the image data information to the cache address specified in the second RDMA cache of the sending process, and transmits the image data information in the second RDMA cache to the first RDMA cache through the network using the sending process through the RDMA data transmission channel.

[0044] In summary, the technical solution disclosed in the present invention establishes an RDMA data transmission channel between the sending process of the client and the receiving process of the server, writes the target data into the cache address of the second data buffer of the sending process, and directly transmits the target data to the first data buffer of the receiving process through the cache address, thereby reducing the number of data copies during the transmission process and reducing the transmission delay of large amounts of data.

[0045] Further, as a refinement and extension of the above embodiment, Figure 3 A data transmission method according to an embodiment of the present disclosure is proposed, which is executed by a client. Figure 1 The embodiment shown, as Figure 3 As shown, and may include the following steps.

[0046] Step 201: Acquire target data to be transmitted.

[0047] For the embodiments of the present disclosure, the implementation process thereof may be specifically referred to the relevant description of step 101 of the embodiment, which will not be repeated here.

[0048] Step 202: Determine the data transmission location information of the RDMA communication connection established with the server, where the data transmission location information includes a first data buffer of the server corresponding to the receiving process, a second data buffer of the client corresponding to the sending process, and an RDMA data transmission channel between the sending process and the receiving process.

[0049] For the implementation process of the embodiment of the present disclosure, please refer to the relevant description of step 102 of the embodiment, which will not be repeated here.

[0050] Step 203: Detect the connection state of the RDMA data transmission channel. If the detected connection state is disconnected, update the connection state of the RDMA data transmission channel to connected by starting a disconnection and reconnection mechanism.

[0051] Among them, the disconnection and reconnection mechanism is to determine whether the RDMA data transmission channel is in a connected state before submitting image data information each time. If it is disconnected (not connected), the reconnection mechanism will be executed. In a specific application scenario, after starting to take pictures, the reconnection mechanism can use the flow rate of the image data submitted after starting to take pictures to judge the frequency of reconnection. The larger the data flow rate, the higher the frequency of starting to reconnect. Conversely, when stopping to take pictures, the data flow rate decreases, and the frequency of starting to reconnect is also low.

[0052] For the disclosed embodiment, the connection status of the RDMA data transmission channel is determined before each transmission of image data. If the connection status is detected to be connected, the target data is written into the second data buffer of the sending process, so that the target data in the second data buffer is transmitted to the first data buffer through the RDMA data transmission channel by the sending process; if the RDMA data transmission channel is detected to be disconnected (i.e., not connected), data interaction is stopped, and the disconnection and reconnection mechanism is started to reconnect the RDMA data transmission channel. Before the RDMA data transmission channel is connected, the receiving thread of the server needs to be started first. After the receiving thread of the server is started, the connection status of the RDMA data transmission channel is updated to connected by receiving the client's request connection signal. As a possible implementation method, the image data information may also include an image data ready signal, which is used to detect the connection status of the RDMA data transmission channel between the server and the client. If the connection status is not connected, the server cyclically waits for the input image data ready signal. Once the ready signal is obtained, the server will start the receiving thread and wait for the client's request connection signal. After the connection is successful, the RDMA data transmission channel status is connected. Before submitting the image data each time, it will be determined whether the RDMA data transmission channel is in a connected state. If disconnected, a reconnection mechanism will be executed. Among them, the image data ready signal is the ready signal carried by the client when sending image data to the server. Once the server obtains the ready signal, the receiving thread will be started to receive the image data information; the request connection signal is a signal sent by the client each time it sends a connection request to the server.

[0053] As a possible implementation method, in order to ensure the efficiency and integrity of data transmission, when the connection state is detected to be disconnected, the data flow of the target data can be detected. When it is determined that the data flow is greater than or equal to the preset threshold, a high-frequency disconnection and reconnection mechanism can be started to update the connection state of the RDMA data transmission channel. The high-frequency disconnection and reconnection mechanism is to send a channel connection request for the RDMA data transmission channel to the server at a high frequency.

[0054] As a possible implementation method, when the connection state is detected to be disconnected, the data flow of the target data can be detected. When it is determined that the data flow is less than a preset threshold, a low-frequency disconnection and reconnection mechanism can be started to update the connection state of the RDMA data transmission channel. The low-frequency disconnection and reconnection mechanism is to send a channel connection request for the RDMA data transmission channel to the server at a low frequency, wherein the first request frequency of the high-frequency disconnection and reconnection mechanism sending the connection request to the server is greater than the second request frequency of the low-frequency disconnection and reconnection mechanism sending the connection request to the server. The preset threshold is a signal value that can be pre-regulated according to actual conditions.

[0055] In a specific application scenario, when shooting, after one shooting is completed, the image data that needs to be transmitted this time will be generated. If the connection is disconnected, the image data cannot be transmitted and the next shooting action cannot be performed. Therefore, in order to ensure the efficiency and quality of image transmission, a controller or system can be used to detect the data flow of the image data. When the detected data flow is greater than or equal to the preset threshold, it means that the shooting is started. At this time, the client will send a request connection signal at a high frequency to request to re-establish the connection; when the detected data flow is less than the preset threshold, it means that the shooting is stopped. At this time, the client will send a request connection signal at a low frequency to request to re-establish the connection. Among them, the preset threshold can be a signal value for starting or stopping shooting. Exemplarily, starting shooting can be represented by 1, stopping shooting can be represented by 0, and the preset threshold can be represented by a value infinitely close to stopping shooting, such as 0.00001, so that the data flow after starting shooting must be greater than or equal to the preset threshold. After starting shooting, the client will send a request connection signal at a high frequency to request to re-establish the connection; when stopping shooting, the data flow is 0, so when the data flow is less than the preset threshold, it means that the shooting is stopped. When stopping shooting, the client will send a request connection signal at a low frequency to request to re-establish the connection.

[0056] Step 204: If it is determined that the transmission status information of the target data sent by the server is successfully transmitted, the target data in the second data buffer is cleared.

[0057] Among them, the transmission status information is the information sent by the server to the client. When the server successfully receives the image data information, it can send a transmission success message to the client; when the server fails to successfully receive the image data information, it can send a transmission failure message to the client. At this time, the client can resend the image data information to the server.

[0058] In a specific application scenario, since the client has only one connection object, there is only one corresponding cache address, while the server can have multiple connection objects, each of which corresponds to a cache address. In this embodiment, only fixed image data will be transmitted each time. After the server receives the image data of each connection object and sends a transmission success message to the client, the cache data of the cache address in the corresponding second RDMA cache of the corresponding connection object will be cleared, and the image data to be transmitted next time will be copied to the cache address in the second RDMA cache.

[0059] Step 205: If it is determined that the transmission status information of the target data sent by the server is a transmission failure, the target data is rewritten into the second data buffer of the sending process, so that the sending process can use the rewritten target data in the second data buffer to transmit it to the first data buffer through the RDMA data transmission channel.

[0060] In a specific application scenario, when the server fails to receive the image data of the connection object, it sends a transmission failure message to the client, and can rewrite the image data information into the cache address in the corresponding second RDMA cache of the connection object corresponding to the client sending process, so that the image data information that should be rewritten in the second RDMA cache can be transmitted to the first RDMA cache of the server through the RDMA data transmission channel by using the sending process.

[0061] In summary, the technical solution disclosed in the present invention establishes an RDMA data transmission channel between the sending process of the client and the receiving process of the server, writes the target data into the cache address of the second data buffer of the sending process, and directly transmits the target data to the first data buffer of the receiving process through the cache address, thereby reducing the number of data copies during the transmission process and reducing the transmission delay of large amounts of data.

[0062] In order to illustrate the data transmission process on the server side, further, this embodiment also provides another data transmission method, which is executed by the server side, such as Figure 4 As shown, the method includes:

[0063] Step 301: Receive target data transmitted by a client using a receiving process.

[0064] Among them, after the client writes the target data into the second data buffer of the sending process, the target data is transmitted to the first data buffer of the receiving process through the RDMA data transmission channel between the sending process and the receiving process by the sending process. The target data can be image data information, text data information, audio data information, etc. In the following embodiments of the present disclosure, the technical scheme of the present disclosure is described by taking the target data as image data information as an example, but it does not constitute a specific limitation of the technical scheme in the present disclosure. In a specific application scenario, the client and the server have established an RDMA communication connection, the client (Windows end) is the sender of the image data, and the server (Linux end, which can be a BCS server) is the receiver of the image data information. A high-speed RDMA network card is used to establish a sending process on the client and a receiving process on the server, and the received image data information is transmitted from the sending process of the client to the receiving process of the server through the RDMA data transmission channel. Accordingly, for the embodiment of the present disclosure, after the client transmits the image data information, the image data information can be transmitted to the first data buffer (i.e., the first RDMA cache) of the server through the RDMA data transmission channel, and accordingly, the server can receive the image data information sent by the client.

[0065] As a possible implementation method, before using the receiving process to receive the target data transmitted by the client, the connection status of the RDMA data transmission channel is first detected. If the connection status is detected to be disconnected, the disconnection and reconnection mechanism is started, and the client continuously sends a request connection signal (i.e., a channel connection request) about the RDMA data transmission channel to the server. After the connection is successful, the RDMA data transmission channel status is connected. Before submitting image data each time, it is determined whether the RDMA data transmission channel is in a connected state. If disconnected, the reconnection mechanism is executed. Accordingly, the steps of the embodiment may specifically include: receiving a channel connection request for the RDMA data transmission channel sent by the client, and in response to the channel connection request, updating the connection status of the RDMA data transmission channel to connected.

[0066] As a possible implementation method, after receiving the target data transmitted by the client using the receiving process, such as Figure 2As shown, the first RDMA cache of the server is used to receive the image data information and the verification information it carries (the verification information can be the size and length of the image data frame, etc.), and the RdmaServer module is used to parse the verification information carried by the image data information, and the parsing result is used to determine whether the received image data is lost. If lost, the transmission status information of the failed transmission is sent to the client, and the image data information is rewritten into the second RDMA cache of the sending process by the retransmission mechanism, and the image data information rewritten in the second RDMA cache is retransmitted to the first RDMA cache through the RDMA data transmission channel by the sending process; if not lost, the transmission status information of the successful transmission is sent to the client, and the RdmaCamera module is used to transmit the image data upward and save it to the memory, wherein the memory can be an external processing memory or a GPU display memory, etc.; the verification information can be the size and length of the image data frame, etc.; the RdmaServer module is mainly used to establish a data transmission channel, data transmission and reception, and management and allocation of resources used for communication; the RdmaCamera module is mainly used to drive interaction with the server and the client.

[0067] Combined with this Figure 2A complete description of the technical solution in the present disclosure is given as follows: the system may include a client (Windows end) and a server (Linux end, which may be a BCS server), wherein the client and the server have created an RDMA communication connection, the client is the sender of the image data, and the server is the receiver of the image data information, a high-speed RDMA network card is used to establish a sending process on the client, and a receiving process is established on the server, and the received image data information is transmitted from the sending process of the client to the receiving process of the server through the RDMA data transmission channel. The specific implementation process is as follows: the client first detects the connection status of the RDMA data transmission channel between the server and the client, and then uses the RdmaCamera module to determine whether there is image data information to be sent in the memory of the optical image acquisition program. If there is image data information to be sent, and the connection status of the RDMA data transmission channel is not connected, the client sends an image data ready signal to the server. Once the server obtains the ready signal, the server will start the receiving thread and wait for the client's request connection signal. After the connection is successful, the RDMA data transmission channel state is connected. Before submitting the image data each time, it will determine whether the RDMA data transmission channel is connected. If it is disconnected, the reconnection mechanism will be executed. If it is in a connected state, verification information is added to the image data information. By writing the verification information into the second RDMA cache, the RdmaClient module of the sending process is used to transmit the image data information written into the second RDMA cache to the first RDMA cache through the RDMA data transmission channel. After the first RDMA cache receives the verification information, the RdmaServer module of the server parses the verification information in the image data information, and determines whether the received image data is lost through the parsing result. If lost, the retransmission mechanism is started, and a transmission failure message is sent to the client; if not lost, the RdmaCamera module is used to transmit the image data upward and save it to the memory, where the memory can be an external processing memory or a GPU video memory, etc., and a transmission success message is sent to the client. The client will clear the cache data corresponding to the cache address in the second RDMA cache and wait for the next image data information to be sent.

[0068] In summary, according to the data transmission method provided by the present invention, an RDMA data transmission channel is established between the sending process of the client and the receiving process of the server, the target data is written to the cache address of the second data buffer of the sending process, and the target data is directly transmitted to the first data buffer of the receiving process through the cache address, thereby reducing the number of data copies during the transmission process and reducing the transmission delay of large amounts of data.

[0069] Based on the above Figure 1 as well as Figure 3The specific implementation of the method shown in this embodiment provides a data transmission device, which is executed by a client, such as Figure 5 As shown, the device includes: an acquisition module 31, a first determination module 32, and a transmission module 33;

[0070] An acquisition module 31, which can be used to acquire target data to be transmitted;

[0071] A first determining module 32, which can be used to determine data transmission information of the RDMA communication connection established with the server, the data transmission information including a first data buffer of the server corresponding to the receiving process, and an RDMA data transmission channel between the sending process and the receiving process;

[0072] The transmission module 33 may be used to write the target data into the second data buffer of the sending process, so as to utilize the sending process to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

[0073] In specific application scenarios, such as Figure 6 As shown, the device further includes: a detection module 34, a first updating module 35;

[0074] A detection module 34, which can be used to detect the connection status of the RDMA data transmission channel;

[0075] The first updating module 35 may be used to update the connection state of the RDMA data transmission channel to connected by starting a disconnection and reconnection mechanism if the connection state is detected to be disconnected.

[0076] In a specific application scenario, the transmission module 33 can be used to write the target data into the second data buffer of the sending process if the connection status is detected to be connected, so as to use the sending process to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

[0077] In a specific application scenario, the first update module 35 can be used to detect the data flow of the target data, determine that the data flow is greater than or equal to a preset threshold, and start a high-frequency disconnection and reconnection mechanism to update the connection state of the RDMA data transmission channel. The high-frequency disconnection and reconnection mechanism is to send a channel connection request for the RDMA data transmission channel to the server at a high frequency.

[0078] In a specific application scenario, the first update module 35 can be used to determine that the data flow is less than a preset threshold, and start a low-frequency disconnection and reconnection mechanism to update the connection status of the RDMA data transmission channel. The low-frequency disconnection and reconnection mechanism is to send a channel connection request for the RDMA data transmission channel to the server at a low frequency, wherein a first request frequency at which the high-frequency disconnection and reconnection mechanism sends a connection request to the server is greater than a second request frequency at which the low-frequency disconnection and reconnection mechanism sends a connection request to the server.

[0079] In specific application scenarios, such as Figure 6 As shown, the device further includes: a second determination module 36;

[0080] The second determination module 36 may be used to determine that the transmission status information of the target data sent by the server is successfully transmitted, and then clear the target data in the second data buffer.

[0081] In specific application scenarios, such as Figure 6 As shown, the device further includes: a third determination module 37;

[0082] If it is determined that the transmission status information of the target data sent by the server is a transmission failure, the target data is rewritten into the second data buffer of the sending process, so that the sending process can use the rewritten target data in the second data buffer to be transmitted to the first data buffer through the RDMA data transmission channel.

[0083] Based on the above Figure 4 The specific implementation of the method shown in this embodiment provides a data transmission device, which is executed by the server, such as Figure 7 As shown, the device may include: a first receiving module 41;

[0084] The receiving module 41 can be used to use the receiving process to receive the target data transmitted by the client, wherein the target data is written by the client into the second data buffer of the sending process, and then the sending process is used to transmit the target data to the first data buffer of the receiving process through the RDMA data transmission channel between the sending process and the receiving process.

[0085] In specific application scenarios, such as Figure 8 As shown, the device further includes: a second receiving module 42 and a second updating module 43;

[0086] The second receiving module 42 may be used to receive a channel connection request regarding an RDMA data transmission channel sent by a client;

[0087] The second updating module 43 may be configured to update the connection state of the RDMA data transmission channel to connected in response to the channel connection request.

[0088] It should be noted that for other corresponding descriptions of the functional units involved in the data transmission device provided in this embodiment, reference can be made to Figure 1 , Figure 3 as well as Figure 4 The corresponding description in will not be repeated here.

[0089] Based on the above Figure 1 , Figure 3 as well as Figure 4 The method shown, and Figure 5 , Figure 6 , Figure 7 as well as Figure 8 The virtual device embodiment shown, in order to achieve the above-mentioned purpose, the embodiment of the present disclosure also provides an electronic device that can be configured in a computer, a mobile phone, a PDA, a gene sequencer, etc., especially a gene sequencer.

[0090] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, sequencers, gene sequencing systems, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0091] Sequencers are used to determine the sequence of the genetic material of a sample. Sequencers can function in a variety of ways and based on a variety of techniques, including sequencing by primer extension using labeled or unlabeled nucleotides, such as sequencing-by-ligation or pyrophosphate sequencing, for example, using any of the Sanger dideoxy method, nanopore or "NexGen" sequencing methods in the art (e.g., using the sequencing platform of MGI, the ROCHE 454 sequencing platform, the ILLUMINA™ SOLEXA™ sequencing platform, the SOLID™ sequencing platform of LIFE TECHNOLOGIES / APPLIED BIOSYSTEMS, the SMRT™ sequencing platform of PACIFICBIOSCIENCES, the POLLONATOR Polony sequencing platform, the COMPLETE GENOMICS sequencing platform, the sequencing platform of INTELLIGENT BIOSYSTEMS, the HELICOS sequencing platform or any other sequencer and system in the art).

[0092] Based on the above Figure 1 , Figure 3 as well as Figure 4The method shown in the embodiment, accordingly, this embodiment also proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above embodiment of the present disclosure as shown in Figure 1 , Figure 3 as well as Figure 4 The method shown.

[0093] Based on the above Figure 1 , Figure 3 as well as Figure 4 The method shown in the embodiment, accordingly, this embodiment also proposes a computer program product, including a computer program, the computer program is executed by a processor in the above embodiment of the present disclosure as shown in Figure 1 , Figure 3 as well as Figure 4 The method shown.

[0094] Based on this understanding, the technical solution of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present disclosure.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the solution of this embodiment, an RDMA data transmission channel is established between the sending process of the client and the receiving process of the server, the target data is written to the cache address of the second data buffer of the sending process, and the target data is directly transmitted to the first data buffer of the receiving process through the cache address, thereby reducing the number of data copies during the transmission process and reducing the transmission delay of large amounts of data.

[0096] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0097] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A data transmission method, characterized in that: The method is executed by a client, and includes: Acquire target data to be transmitted; Determine data transmission location information of an RDMA communication connection established with a server, the data transmission location information including a first data buffer of a receiving process corresponding to the server, a second data buffer of a sending process corresponding to the client, and an RDMA data transmission channel between the sending process and the receiving process; The target data is written into the second data buffer of the sending process, so that the sending process is used to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

2. The method according to claim 1, characterized in that Before writing the target data into the second data buffer of the sending process, the method further includes: Detecting the connection status of the RDMA data transmission channel; If the connection state is detected to be disconnected, the connection state of the RDMA data transmission channel is updated to connected by starting a disconnection and reconnection mechanism; The step of writing the target data into the second data buffer of the sending process comprises: If the connection state is detected to be connected, the target data is written into a second data buffer of the sending process, so that the sending process is used to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

3. The method according to claim 2, characterized in that The updating the connection state of the RDMA data transmission channel to connected by starting the disconnection and reconnection mechanism includes: Detecting data flow of the target data; Determine that the data flow is greater than or equal to a preset threshold, and start a high-frequency disconnection and reconnection mechanism to update the connection status of the RDMA data transmission channel, wherein the high-frequency disconnection and reconnection mechanism is to send a channel connection request for the RDMA data transmission channel to the server at a high frequency.

4. The method according to claim 2, characterized in that: The updating the connection state of the RDMA data transmission channel to connected by starting the disconnection and reconnection mechanism includes: Determine that the data flow is less than the preset threshold, start a low-frequency disconnection and reconnection mechanism to update the connection state of the RDMA data transmission channel, the low-frequency disconnection and reconnection mechanism is to send a channel connection request for the RDMA data transmission channel to the server at a low frequency, wherein a first request frequency of the high-frequency disconnection and reconnection mechanism sending the connection request to the server is greater than a second request frequency of the low-frequency disconnection and reconnection mechanism sending the connection request to the server.

5. The method according to claim 1, characterized in that The method further comprises: If it is determined that the transmission status information of the target data sent by the server is successfully transmitted, the target data in the second data buffer is cleared.

6. The method according to claim 5, characterized in that The method further comprises: If it is determined that the transmission status information of the target data sent by the server is a transmission failure, the target data is rewritten into the second data buffer of the sending process, so that the target data rewritten in the second data buffer is transmitted to the first data buffer through the RDMA data transmission channel by the sending process.

7. A data transmission method, characterized in that: The method is executed by the server, and includes: A receiving process is used to receive target data transmitted by a client, wherein the target data is written by the client into a second data buffer of a sending process, and then the sending process is used to transmit the target data to a first data buffer of the receiving process through an RDMA data transmission channel between the sending process and the receiving process.

8. The method according to claim 7, characterized in that Before using the receiving process to receive the target data transmitted by the client, the method further includes: Receiving a channel connection request for the RDMA data transmission channel sent by a client; In response to the channel connection request, the connection state of the RDMA data transmission channel is updated to connected.

9. A data transmission device, characterized in that: The device is executed by a client, and includes: An acquisition module, used for acquiring target data to be transmitted; A determination module, used to determine data transmission information of an RDMA communication connection established with a server, wherein the data transmission information includes a first data buffer of a receiving process corresponding to the server, and an RDMA data transmission channel between the sending process and the receiving process; The transmission module is used to write the target data into the second data buffer of the sending process, so as to utilize the sending process to transmit the target data in the second data buffer to the first data buffer through the RDMA data transmission channel.

10. A data transmission device, characterized in that: The device is executed by a server, and includes: The receiving module is used to use the receiving process to receive the target data transmitted by the client, wherein the target data is written by the client into the second data buffer of the sending process, and then the target data is transmitted by the sending process to the first data buffer of the receiving process through the RDMA data transmission channel between the sending process and the receiving process.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7 or the method of any one of claims 8 to 10.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 10.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 10.

Citation Information

Cited By

  • Data processing method, device, storage medium and computer program product

    WO2026138562A1