Data transmission method, device, equipment and readable storage medium

Abstract

The present application discloses a data transmission method, device, equipment and readable storage medium, which obtains a data block acquisition request sent by a target node, the data block acquisition request carries a target data block identifier; wherein the target node and the local node belong to different service nodes in a distributed system, each service node in the distributed system stores different data blocks, and each service node is registered with meta information of each data block, and a corresponding communication queue pair is pre-created between every two different service nodes in the distributed system; the target meta information associated with the target data block identifier is obtained, and the target data block associated with the target meta information is searched from the local storage; the target communication queue pair pre-created by the local node for the target node is determined, and the target data block is loaded into the target communication queue pair; the target data block in the target communication queue pair is transmitted to the target node through a remote direct memory access network. The reliability of data transmission is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transmission method, apparatus, device and readable storage medium. Background Art

[0002] With the rapid development of information technology, business services have been diversified, which means that business services involve the application of a large amount of data. That is, business services need to frequently obtain data for business processing during operation. Therefore, service nodes need to transfer stored data to computing components for calculation. At the same time, they also need to communicate the data to be transmitted. During this period, a large amount of data communication and concurrent execution of data calculation will be involved. That is, service nodes need to run both computing processes and communication processes.

[0003] Therefore, when faced with processing large amounts of data, a large number of computing processes and communication processes will occupy a large amount of computing resources of the service node, such as a large number of thread resources, which will cause the load to be too high, easily leading to data transmission failures and affecting the normal operation of business services. Summary of the invention

[0004] The present application provides a data transmission method, apparatus, device and readable storage medium, which can improve the reliability of data transmission and improve the operational stability of business services.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] The present application provides a data transmission method, including:

[0007] Obtaining a data block acquisition request sent by a target node, wherein the data block acquisition request carries a target data block identifier;

[0008] The target node and the local node belong to different service nodes in the distributed system, different data blocks are stored on each service node in the distributed system, and meta information of each data block is registered on each service node, and a corresponding communication queue pair is pre-created between every two different service nodes in the distributed system;

[0009] Acquire target meta information associated with the target data block identifier, and search for the target data block associated with the target meta information from local storage;

[0010] Determine a target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair;

[0011] The target data block in the target communication queue pair is transmitted to the target node through a remote direct memory access network.

[0012] Accordingly, an embodiment of the present application provides a data transmission device, including:

[0013] An acquisition unit, configured to acquire a data block acquisition request sent by a target node, wherein the data block acquisition request carries a target data block identifier;

[0014] The target node and the local node belong to different service nodes in the distributed system, different data blocks are stored on each service node in the distributed system, and meta information of each data block is registered on each service node, and a corresponding communication queue pair is pre-created between every two different service nodes in the distributed system;

[0015] A search unit, configured to obtain target meta information associated with the target data block identifier, and search for the target data block associated with the target meta information from a local storage;

[0016] a loading unit, configured to determine a target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair;

[0017] The transmission unit is used to transmit the target data block in the target communication queue pair to the target node through a remote direct memory access network.

[0018] In some implementations, the search unit is further configured to:

[0019] Determine, according to the target meta information, a target storage node address corresponding to the target data block identifier;

[0020] When it is identified that the target storage node address is consistent with the node address of the local node, the target data block corresponding to the target data block identifier is searched from the local storage.

[0021] In some implementations, the search unit is further configured to:

[0022] When it is identified that the target storage node address is inconsistent with the node address of the local node, current time information is acquired, and data block exception information is generated according to the current time information and the node address of the local node;

[0023] Feedback the data block exception information to the target node.

[0024] In some embodiments, the loading unit is further used for:

[0025] The target data block in the local storage is transferred to the main storage space of the local node, where the main storage space is used to store data directly read and written by the local processor;

[0026] The target data block in the main storage space is copied to the target communication queue pair.

[0027] In some implementations, the target communication queue pair includes a target sending queue and a target receiving queue, and the loading unit is further configured to:

[0028] The target data block in the main storage space is copied to the target sending queue in the target communication queue pair.

[0029] In some implementations, the transmission unit is further configured to:

[0030] Run the direct transfer function corresponding to the remote direct memory access network;

[0031] The target data block in the target communication queue pair is transmitted to the target node through the direct transmission function.

[0032] In some embodiments, the data transmission device further includes a detection unit, configured to:

[0033] Get the current transmission delay and packet loss rate of the remote direct memory access network;

[0034] The transmission unit is further used for:

[0035] When the transmission delay duration is less than a preset delay threshold and the packet loss rate is less than a preset packet loss rate threshold, the target data block in the target communication queue pair is transmitted to the target node via the remote direct memory access network.

[0036] In some implementations, the transmission unit is further configured to:

[0037] When the transmission delay time is greater than or equal to a preset delay threshold, or the packet loss rate is greater than or equal to a preset packet loss rate threshold, acquiring the network address of the target node;

[0038] Establishing a data transmission channel between the local node and the target node of the network address through a socket interface;

[0039] When the target data block is found in the main storage space of the local node, the target data block in the main storage space is transmitted to the target node corresponding to the network address through the data transmission channel;

[0040] When the target data block is not found in the main storage space, the target data block is loaded from the local storage into the main storage space, and the target data block in the main storage space is transmitted to the target node corresponding to the network address through the data transmission channel.

[0041] In some embodiments, the data transmission device further includes a starting unit, configured to:

[0042] Running the remote task program on the local node;

[0043] The local node running the remote task program and other service nodes running the remote task program together form a distributed system;

[0044] Starting a local external communication service process on the local node after running the remote task program;

[0045] Through the local external communication service process, communication queue pairs are respectively created with other service nodes in the distributed system.

[0046] In some embodiments, the startup unit is further used to:

[0047] Acquiring communication configuration parameters of the local node;

[0048] Allocating a target amount of memory resources to the local external communication service process according to the communication configuration parameters;

[0049] Obtaining a service node address of each other service node in the distributed system;

[0050] Based on the target amount of memory resources, a communication queue pair between the local node and each service node address is respectively created in the local external communication service process.

[0051] In some implementations, the metadata of the data block is registered in an external communication service process running on each service node, and the search unit is further configured to:

[0052] According to the target data block identifier, corresponding target meta information is searched from the local external communication service process of the local node.

[0053] In some embodiments, the data transmission device further includes a registration unit, configured to:

[0054] Obtaining meta information of a data block pre-stored by each service node in the distributed system for a target business service, the meta information including a data block identifier and a storage node address;

[0055] Each piece of meta information is registered in the local external communication service process, so that the local external communication service process records the storage node address associated with each data block identifier.

[0056] In some embodiments, the data transmission device further includes a data block acquisition unit, configured to:

[0057] Determine the business data block identifier obtained by the local node for the target business service request;

[0058] Based on the local external communication service process, query the remote node address corresponding to the business data block identifier;

[0059] Sending a service data block acquisition request to a target service node corresponding to the remote node address in the distributed system, wherein the data block acquisition request carries a service data block identifier;

[0060] The business data block corresponding to the business data block identifier fed back by the target service node is obtained through the target receiving queue in the communication queue pair pre-created between the local node and the target service node, so as to run the target business service according to the business data block.

[0061] In addition, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above-mentioned data transmission method is implemented when the processor executes the computer program.

[0062] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for loading by a processor to execute the above-mentioned data transmission method.

[0063] In addition, the embodiment of the present application also provides a computer program product, the computer program product includes a computer program, and the computer program can be stored in a storage medium. The processor of the computer device reads the computer program from the storage medium and executes the computer program to implement the above data transmission method.

[0064] In an embodiment of the present application, a data block acquisition request sent by a target node is obtained, and the data block acquisition request carries a target data block identifier; wherein the target node and the local node belong to different service nodes in a distributed system, and different data blocks are stored on each service node in the distributed system, and meta-information of each data block is registered on each service node, and a corresponding communication queue pair is pre-created between every two different service nodes in the distributed system; the target meta-information associated with the target data block identifier is obtained, and the target data block associated with the target meta-information is searched from the local storage; the target communication queue pair pre-created by the local node for the target node is determined, and the target data block is loaded into the target communication queue pair; the target data block in the target communication queue pair is transmitted to the target node through a remote direct memory access network.

[0065] In the embodiment of the present application, a distributed system corresponding to a big data processing framework is formed by multiple service nodes, and each service node is registered with the meta information of each data block stored on all service nodes. According to the meta information of each data block, the storage node address corresponding to each data block in the distributed system can be quickly located, so that there is no need to traverse and send data block acquisition requests to all service nodes in the distributed system, reducing network interactions and improving data acquisition efficiency. In addition, based on the remote direct memory access network, the data block can be remotely transmitted to other service nodes, thereby realizing the separation of data calculation and communication, avoiding the simultaneous operation of the calculation process and the communication process on a single service node, which makes the operation load of the service node too high, thereby avoiding the risk of data transmission failure and improving the reliability of data transmission. At the same time, a communication queue pair is established separately between any two service nodes in the distributed system of the big data processing framework, and the communication queue pair can be reused for multiple data block transmissions between the two service nodes, avoiding frequent full connections between any two service nodes. In this way, while satisfying reliable transmission in big data processing, communication delays are further avoided, thereby improving data block transmission efficiency. In addition, when separating data calculation and communication, the computing resources of the currently used service nodes are utilized, and each service node can be used as both a computing end and a communication end, so there is no need to set up an additional communication server, reducing equipment costs.

[0066] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 A schematic diagram of a data transmission system according to an embodiment of the present invention;

[0069] Figure 2 A schematic diagram of the steps of the data transmission method provided in the embodiment of the present application;

[0070] Figure 3 A schematic diagram of another step flow chart of the data transmission method provided in an embodiment of the present application;

[0071] Figure 4 An example diagram of the architecture of a data transmission scenario provided in an embodiment of the present application;

[0072] Figure 5 An example diagram of data block transmission effect in a data transmission scenario provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of the structure of a data transmission device provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0077] It can be understood that in the specific implementation of the present application, related data such as data blocks, data block identifiers, storage node addresses, metadata, etc., when the above embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.

[0078] In addition, when the embodiment of the present application needs to obtain data blocks, data block identifiers, storage node addresses, meta-information and other related data, a separate license or separate consent for the data blocks, data block identifiers, storage node addresses, meta-information and other related data will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining a separate license or separate consent for the data blocks, data block identifiers, storage node addresses, meta-information and other related data, the necessary data blocks, data block identifiers, storage node addresses, meta-information and other related data required to enable the normal operation of the embodiment of the present application will be obtained.

[0079] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0081] The embodiments of the present application provide a data transmission method, apparatus, device, and readable storage medium. Specifically, the data transmission method of each embodiment of the present application can be executed by a computer device, which can be a server or a terminal or other device. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Among them, the terminal can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart home appliance, a car terminal, an intelligent voice interaction device, an aircraft, etc., but is not limited to this.

[0082] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0083] Remote Direct Memory Access (RDMA) is a network communication technology that allows a computer to directly access the memory of another computer through the network without the intervention of the CPU of the local node or the remote node, thereby achieving high-throughput and low-latency data transmission. Remote direct memory access can support remote direct data access, which is created to solve the delay of server-side data processing in network transmission. RDMA transfers information and data directly to the computer's storage area through the network, quickly moving data from one system to the remote system memory without any impact on the operating system, so that little computer processing function is not required. It eliminates the overhead of external memory copying and context switching, thereby freeing up memory bandwidth and CPU cycles to improve application system performance.

[0084] Transmission Control Protocol (TCP) is a connection-oriented, reliable, byte-stream-based transport layer communication protocol. TCP is designed to adapt to the layered protocol hierarchy that supports multiple network applications. Pairs of application processes in an interconnected computer communication network can rely on TCP to provide reliable communication services to transmit byte streams. TCP supports bidirectional data flows, and applications can also send data in only one direction. Between hosts, TCP uses port numbers to identify application services and can transmit data streams in multiple channels.

[0085] The communication queue pair is an important concept in RDMA (Remote Direct Memory Access) network communication, which is used to establish a communication connection and perform data transmission between two nodes. It actually consists of two queues, one for sending operations (Send Queue) and the other for receiving operations (Receive Queue).

[0086] A data transmission method provided by an embodiment of the present application can use existing service nodes to form a distributed system, and an external communication service process is respectively run on each service node in the distributed system, and the external communication service process on each service node is registered with the metadata of all data blocks in the distributed system. The storage node address of the data block can be determined through the metadata of the data block. Any service node can determine the storage node address of each data block in the distributed system based on the metadata registered by the built-in external communication service process. When a target data block is needed, a data block acquisition request can be sent to the service node with the storage node address, so as to obtain the target data block from the service node through a remote direct memory access network based on a communication queue pair between two service nodes, thereby transmitting the data block. Please continue to refer to the following specific embodiments for details.

[0087] For example, see Figure 1 , is a scenario diagram of a data transmission system provided in an embodiment of the present application, the system includes a terminal 110 and a server 120.

[0088] Among them, the terminal 110 may be installed with a target application, and the corresponding application service may be run through the target application. The target application may be collectively referred to as a client, and the terminal 110 may send a service request to the server 120. The server 120 may be a service server corresponding to the target application, for example, the server 120 may be a distributed system composed of multiple service nodes, or may be a service node in the distributed system.

[0089] Among them, the server 120 executes the steps of the data transmission method, specifically, it can obtain a data block acquisition request sent by the target node, the data block acquisition request carries a target data block identifier; wherein the target node and the local node belong to different service nodes in the distributed system, each service node in the distributed system stores different data blocks, and each service node is registered with metadata of each data block, and a corresponding communication queue pair is pre-created between every two different service nodes in the distributed system; obtain the target metadata associated with the target data block identifier, and search for the target data block associated with the target metadata from the local storage; determine the target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair; transmit the target data block in the target communication queue pair to the target node through a remote direct memory access network.

[0090] Thereafter, the target node may perform data calculation according to the target data block to obtain a data calculation result, and push data to the client on the terminal 110 according to the data calculation result.

[0091] Based on the above, the present application forms a distributed system corresponding to the big data processing framework through multiple service nodes, and each service node is registered with the metadata of each data block stored on all service nodes. According to the metadata of each data block, the storage node address corresponding to each data block in the distributed system can be quickly located, so that there is no need to traverse and send data block acquisition requests to all service nodes in the distributed system, reducing network interactions and improving data acquisition efficiency. In addition, based on the remote direct memory access network, the data block can be remotely transmitted to other service nodes, thereby realizing the separation of data calculation and communication, avoiding the simultaneous operation of the calculation process and the communication process on a single service node, which makes the operation load of the service node too high, thereby avoiding the risk of data transmission failure and improving the reliability of data transmission. At the same time, a communication queue pair is established separately between any two service nodes in the distributed system of the big data processing framework, and the communication queue pair can be reused for multiple data block transmissions between the two service nodes, avoiding frequent full connections between any two service nodes. In this way, while satisfying reliable transmission in big data processing, communication delays are further avoided, thereby improving data block transmission efficiency. In addition, when separating data calculation and communication, the computing resources of the currently used service nodes are utilized, and each service node can be used as both a computing end and a communication end, so there is no need to set up an additional communication server, reducing equipment costs.

[0092] It should be noted that the above are only examples and can also be applied to other data transmission scenarios, which will not be described in detail here.

[0093] For ease of understanding, each step of the data transmission method will be described in detail below. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0094] See also Figure 2 , Figure 2 A schematic diagram of a step flow chart of a data transmission method provided in an embodiment of the present application. In an embodiment of the present application, a computer device executes the data transmission method, such as Figure 1 The execution of the server 120 in the example is used for explanation, and the specific process is as follows:

[0095] 101. Obtain a data block request sent by a target node.

[0096] With the development of information technology, business services have become diversified, such as advertising push, news push, short video playback and other business services. The operation of these business services generally involves data processing. The data processing process can be understood as data query to process the queried data, that is, the service node transfers the stored data to the computing component for calculation. This process includes data communication and data calculation. Therefore, the service node needs to run the computing process and the communication process to run the corresponding business services.

[0097] However, the operation of business services involves the application of a large amount of data, that is, the business service needs to frequently obtain a large amount of data for calculation and processing during operation, during which data communication and calculation need to be frequently performed. The communication process and the calculation process are executed in sequence. If there are a large number of calculation processes and communication processes on the service node, a large amount of computing resources of the service node will be occupied. For example, a large amount of thread resources will be occupied, making the thread load of the service node too high, which can easily lead to data transmission failure and affect the normal operation of the business service.

[0098] In response to the above problems, the embodiment of the present application is based on the current multiple service nodes to perform initialization to run the program corresponding to the big data processing framework to create a corresponding distributed system for the business service. Each service node in the distributed system has independent computing capabilities, and each service node runs an independent external communication service process. At the same time, a service node generally only serves as a computing node, and another service node serves as a communication node, thereby realizing independent separation of the communication process and the computing process during the operation of the business service, avoiding the problem of excessive load on a single service node due to the running of the computing process and the communication process on the same node at the same time, reducing the risk of data transmission failure, improving data transmission reliability, and ensuring the stable operation of the service node. In this way, the stable operation of the business service is maintained through the joint collaboration of multiple service nodes in the distributed system.

[0099] In the embodiment of the present application, the computer device may be a distributed system (i.e. Figure 1 Any service node in the server 120 executes the data transmission method to achieve reliable data transmission and maintain the normal operation of the business service. Among them, the business service can be advertising push, news and current events push, short video playback, online shopping pages, instant messaging and other services involving big data processing, which are not listed here one by one.

[0100] In the embodiment of the present application, a business service can be jointly operated based on multiple service nodes in a distributed system, wherein data blocks related to the business service are distributedly stored in the distributed system. At the same time, a target node in the distributed system is used as a computing node, and other service nodes in the distributed system can be used as communication nodes. The communication node stores the data blocks required by the target node. When the target node sends a data block acquisition request to other service nodes, the external communication service process (i.e., the communication process) running on the communication node is used to send the locally stored target data block to the target node. Specifically, for example, the target node, as the computing end of the business service, sends a data block acquisition request to the local node (a service node other than the target node in the distributed system), and the data block acquisition request carries the target data block identifier, in the hope of obtaining the target data block required by the target data block identifier from the local node. In this way, the communication of data blocks is realized while the computing process and the communication process are separated independently, thereby improving the system performance, thereby improving the reliability of data transmission, ensuring that the required data blocks can be provided for the operation of the business items of the business service, and ensuring the stability of the business service.

[0101] Among them, the target node and the local node belong to different service nodes in the distributed system. Different data blocks are stored on each service node in the distributed system, and the metadata of each data block is registered on each service node. A corresponding communication queue pair is pre-created between every two different service nodes in the distributed system.

[0102] It should be noted that each business item can be understood as a data processing. For example, taking advertising push as an example, executing an advertising push to a target object can be regarded as a data processing. Each data processing can be understood as the need to perform at least one data communication and data calculation, that is, the target node responsible for processing the business item needs to send a data block acquisition request to another service node in the distributed system to obtain the target data block from other service nodes in the distributed system. Since business services are generally open to a wide range of objects, multiple business items may be involved at the same time, and a business item may involve one or more data communications, that is, it involves big data processing, and the "distributed system" of the embodiment of the present application belongs to the system under the big data processing framework, through the cooperation between multiple service nodes in the distributed system to support the big data processing corresponding to the business service, to ensure the normal operation of the business service. It should be noted that at the same time, there can be multiple service nodes acting as computing nodes in the distributed system, and there can be multiple service nodes acting as communication nodes. The number of specific computing nodes and communication nodes can be determined according to actual conditions, and is not limited here.

[0103] Among them, data blocks are the basic data units in computer storage systems and the smallest logical units for data read and write operations on storage devices such as disks. For example, data will be organized into data blocks for storage and management. Usually, a data block contains a certain number of bytes, representing an independent piece of data, and providing parameters for data processing of business services.

[0104] Among them, the data block acquisition request carries a target data block identifier, which can be a unique identifier in a distributed system, indicating the identity information of the corresponding data block, and is used to accurately distinguish and locate the corresponding data block. Specifically, different data blocks are stored on each service node in the distributed system, and any two data blocks in the distributed system are different. In order to distinguish different data blocks in the distributed system, they can be represented by an identifier, and the identifiers of the data blocks are different. When assigning a data block identifier to each data block, they can be numbered in sequence according to the order in which the data blocks are generated, and the number is used as the data block identifier of the corresponding data block. A hash calculation can also be performed according to the generation time of the data block and the value of the byte size occupied by the data block, and the hash value is used as the data block identifier of the corresponding data block. In addition, a data block identifier can be generated randomly and non-repetitively for each data block. The above is only an example and is not limited here.

[0105] Among them, the metadata of the data block can be understood as the index information of the data block in the distributed system. Each data block has a corresponding metadata, which can include the data block identifier and storage node address of the corresponding data block. The storage node address can specifically be the physical location address of the corresponding service node, the service node identifier (such as the independent number and serial number of each service node in the distributed system), the Internet Protocol address, etc., which is not limited here. It should be noted that the metadata of all data blocks corresponding to the business data in the entire distributed system is registered on each service node in the distributed system, so that each service node in the distributed system can locate the node position of each data block based on the metadata of the locally registered data block.

[0106] Among them, the communication queue pair can be a communication channel specially used for data transmission between two specific service nodes in a distributed system, which can include a sending queue and a receiving queue, that is, a communication queue pair is composed of a sending queue and a receiving queue, the sending queue is used to store data blocks that need to be sent, and the receiving queue can be used to receive data blocks transmitted from other service nodes. It should be noted that for a service node in a distributed system, it is necessary to create a separate communication queue pair with any other service node in the system, that is, a service node needs to independently establish a communication queue pair with any other service node. In this way, if the distributed system contains multiple service nodes, each service node usually has multiple communication queue pairs, and the number of communication queue pairs owned by each service node is usually the total number of service nodes contained in the distributed system minus one.

[0107] It should be noted that the service node at the sending end puts the data block into the sending queue so that the data block can be directly transmitted to the receiving queue of the service node at the receiving end. The service node at the receiving end then obtains the data block from the receiving queue of the communication queue pair pre-created for the service node at the sending end. In this way, efficient data transmission is achieved, the copying and movement of data in the operating system kernel buffer is reduced, the communication delay is reduced, and the bandwidth utilization is improved. It should be noted that in addition to being used to transmit data blocks, the communication queue pair can also be used to send and receive "requests" between service nodes. This is not limited here.

[0108] In an embodiment of the present application, the system performance is improved by separating the computing process and the communication process to improve the reliability of data transmission. Compared with the tight coupling of computing and communication in traditional big data processing, which easily leads to resource contention and performance bottlenecks, the embodiment of the present application starts an independent external communication service process to separate communication-related operations (such as data transmission, network connection management, etc.) from the computing process, so that the two can run independently without interfering with each other. In this way, the computing process focuses on the execution of data processing algorithms, and the communication process specializes in network communication transactions, which improves the modularity of the system and facilitates maintenance management and function expansion. For example, in large-scale data analysis tasks, computing nodes can efficiently process data calculations, and communication processes ensure high-speed and reliable data transmission, avoid affecting computing efficiency due to communication delays or failures, optimize system resource utilization, and improve overall performance.

[0109] In order to separate the computing process from the communication process, during the initialization phase of the business service, it is necessary to run a program related to the big data processing framework for the business service on multiple service nodes to form a distributed system corresponding to the big data processing framework. Furthermore, a communication service process is externalized on each service node to realize the externalization of the communication process. At the same time, the service nodes participating in the data processing of the business service in the distributed system can only play the role of computing node or communication node; in addition, any two service nodes have a communication queue pair dedicated to communicating with each other. These multiple service nodes cooperate with each other in the big data processing framework to provide direct access and reading of the data blocks of the business service, supporting the normal operation of the business service.

[0110] In some implementations, during the business service initialization phase, a remote task program may be run on a local node to form a distributed system with other service nodes, and a communication queue pair may be established with each other service node through the running local external communication service process. For example, before step 101, the following may also be included:

[0111] (A.1) Run the remote task program on the local node;

[0112] Among them, the local node after running the remote task program and other service nodes running the remote task program together form a distributed system;

[0113] (A.2) Starting a local external communication service process on the local node after running the remote task program;

[0114] (A.3) Through the local external communication service process, create communication queue pairs with other service nodes in the distributed system.

[0115] The local node may be any service node.

[0116] The remote task program can be understood as a program of the big data processing framework of the business service, which is a tool for processing large-scale data in a distributed system, including but not limited to functions such as data storage, computing, communication, analysis, and process orchestration. Multiple service nodes run the remote task program respectively to form a distributed system of the big data processing framework for business services.

[0117] It should be noted that the role of the remote task program includes at least three stages of data processing in business services: In the data block registration stage, the remote task program is responsible for marking the data block's globally unique identifier (i.e., data block identifier), ensuring the consistency and uniqueness of the data block's metadata, and synchronizing it to each service node in the distributed system, laying the foundation for subsequent data positioning and transmission. In the data block acquisition request initiation stage, the remote task program on the target node generates a request instruction based on the task requirements, triggers the communication process, and coordinates the data transmission operation between the target node and the local node, including allocating worker threads, managing thread communication and data transmission status, etc. In the data processing stage, the remote task program on the target node organizes computing node resources, schedules and executes data processing tasks (such as data analysis and mining algorithms), and collaborates with the communication framework to achieve efficient coordination of computing and communication, ensuring the smooth flow and processing of data between each service node, and is a key supporting component for achieving reliable processing and efficient communication of big data.

[0118] Among them, the external communication service process can be a service process that runs independently of the computing process and is specifically responsible for handling communication transactions between service nodes. Specifically, in a distributed system, the external communication service process, as an independent functional module, focuses on network communication-related operations, such as data transmission, connection management, communication protocol processing, etc., to ensure that data is accurately and efficiently exchanged between nodes, and cooperates with the computing process to improve the overall performance and reliability of the distributed system. It should be noted that the external communication service process is the key to realizing the separation of computing and communication architecture. By running independently on a service node in a distributed system (such as a "local node" from the perspective of the embodiment of the present application), the workload of the computing process in the service node is effectively reduced, and the communication operation is avoided. Contention for computing resources on the service node allows the service node computing process to focus on the core task of data processing and improve computing efficiency. The function of pre-establishing a communication queue pair is to accelerate data transmission, reduce communication delays, and improve the overall communication performance of the system. It works closely with the programs of the big data processing framework to implement functions such as data block metadata registration and management, data block acquisition request response, and data transmission control, ensuring that the communication link in the big data processing process is reliable and efficient. It is an important component for ensuring the high reliability and high performance of the big data processing system, and provides strong support for the stable operation of big data applications in complex network environments.

[0119] Specifically, in order to separate the computing process from the communication process, first, the same remote task program is run on each service node, so that the multiple service nodes running the remote task program together constitute the distributed system corresponding to the big data processing framework. Then, each service node in the distributed system also needs to run an external communication service process, which is the basis for the subsequent separation of the computing process and the communication process on the physical service node, effectively reducing the workload of the computing process, avoiding the contention of the communication operation for computing resources, and making the computing process focus on the core tasks of data processing, improving the efficiency of data processing, and improving the response speed of business services. Finally, in the external communication process running on each service node, a communication queue pair must be created separately for any other service node in the distributed system. In this way, the external communication service process running on each service node contains multiple communication queue pairs for communication with different other service nodes. In this way, the pre-established communication queue pair function accelerates data transmission, reduces communication delays, and improves the overall communication performance of the system. It works closely with the big data processing framework program to ensure that the communication link in the big data processing process is reliable and efficient. It is an important part of ensuring the high reliability and high performance of the big data processing system, and provides strong support for the stable operation of big data applications in complex network environments.

[0120] In some embodiments, each service node in the distributed system may first determine the amount of available resources allocated to each communication process, and obtain the address of each service node in the distributed system, and create a communication queue pair for data communication with each service node based on the available resources. For example, step (A.3) may include: obtaining communication configuration parameters of the local node; allocating a target amount of memory resources to the local external communication service process according to the communication configuration parameters; obtaining the service node address of each other service node in the distributed system; and creating a communication queue pair between the local node and each service node address in the local external communication service process based on the target amount of memory resources.

[0121] Among them, the communication configuration parameters can be the configuration information of the local service node for the communication function, which may include but is not limited to the communication queue pair attribute configuration structure, and the communication queue pair attribute configuration structure is used to set the depth of the communication queue pair, the type of sending queue and receiving queue, and / or service quality parameters, etc.

[0122] The memory resource may be a RAM resource, which is a component used to temporarily store the programs and data being executed. It can only temporarily store data for exchanging cache data with the central processing unit. The size of the RAM directly determines how many programs the system can run. The larger the RAM, the faster the system runs programs.

[0123] Among them, the service node address can be the address corresponding to the corresponding service node. It should be noted that each service node is an individual device with independent computing capabilities. In order to accurately obtain data blocks between service nodes and distinguish between service nodes in a system where computing and communication are separated, the service node address can be used to represent the identity information of the service node. Each service node address needs to be registered, declared or broadcasted on the Internet so that any device unit can search for the service node. Among them, the service node address can be the physical address of the corresponding service node, such as "XX Province XX District XX Street XX Road XX No. XX", or a custom identification character. In addition, it can also be an Internet Protocol address. The above is only an example and is not limited here.

[0124] It should be noted that, in the case of separating computing and communication, in order to directly and quickly obtain data blocks from remote service nodes, the remote direct memory access (RDMA) network is used to quickly transmit data blocks to ensure the normal operation of business services. Based on this, each service node can start the queue pair creation process through the relevant configuration acquisition function in the interface library of the remote direct memory access network. Taking the local service node as an example, the local service node obtains the local communication configuration parameters through this function, so as to allocate corresponding memory resources for the communication process in combination with the local communication configuration parameters, which can be specifically running memory resources. The running memory resources allocated by the local service node for communication are mainly used to run the local external communication service process to meet the purpose of remote and rapid transmission of data blocks. Therefore, the local service node can create a communication queue pair for each other service node in the distributed system in the external communication service process running in the local service node based on the allocated running memory resources, so as to reduce the communication delay of the data block, improve the overall communication performance of the system, and promote the stable operation of business services.

[0125] Exemplarily, the local node can obtain the communication configuration related parameters of the local service node through the configuration acquisition function (such as the "ibv_create_qp()" function), including the node communication context, the queue pair attribute configuration structure (used to set the queue depth, the send and receive queue type, the service quality parameters, etc.) and the communication address information of the target node. Then, the driver allocates memory resources according to the communication configuration parameters and builds the queue data structure. Initialize the send and receive queue buffer, pointer and control block, and establish a mapping association with the hardware queue. Then, the local service node registers the information of the created communication queue pair (including identifiers, attribute parameters, memory address mapping relationships, etc.) to the resource management module of the local remote direct memory access network, and exchanges information with the target node through the remote direct memory access network to negotiate communication parameters such as flow control window and maximum transmission unit to reach an agreement. In this way, the creation and initialization of the queue pair is completed, so as to establish an efficient communication link between the two service nodes, ensuring that data can be transmitted at high speed and reliably according to the characteristics of remote direct memory access (RDMA).

[0126] Through the above method, when multiple service nodes start running the corresponding remote task programs in the business service initialization phase, these multiple service nodes can form a distributed system corresponding to a big data processing framework, and each service node in the distributed system runs an external communication service process, and the service nodes independently create communication queue pairs to realize data communication operations between service nodes under the condition of separation of computing and communication. Based on this, any service node in the distributed system can receive data block acquisition requests from any other service node (i.e., target node), and at the same time, it can also send data block acquisition requests to other service nodes, thereby requesting to obtain data blocks from other service nodes, realizing separation of computing and communication, and improving subsequent data computing efficiency.

[0127] 102. Obtain target meta information associated with the target data block identifier, and search for the target data block associated with the target meta information from local storage.

[0128] In an embodiment of the present application, after receiving a data block acquisition request sent by any target node in the distributed system, the local service node can search for the corresponding target metadata in the local service node according to the target data block identifier carried in the data block acquisition request, and thus search for the corresponding target data block according to the target metadata. In this way, when computing and communication are separated, preliminary preparations can be made for the subsequent delivery of target data blocks, thereby improving the transmission efficiency of data blocks.

[0129] Among them, the target meta information can be understood as the index information of the target data block, which is not limited to including the target data block identifier and storage node address. The description of the target meta information can refer to the previous description of "meta information of data block", which will not be repeated here.

[0130] Among them, local storage can be storage space on a local service node, such as a hard disk, a disk, a cache, a main memory, etc., which is not limited here.

[0131] Among them, the target data block is the data unit that needs to be searched currently and needs to be remotely transmitted later. It can be target business data, files, tables, operation functions, codes, pictures, texts, numbers, etc., which are not limited here.

[0132] It should be noted that, when computing and communication are separated, in order to enable a service node to accurately request and obtain a target data block from another service node, corresponding meta information is stored on each service node for all data blocks stored on all service nodes in the distributed system, for example, it can be stored in a specific file, in a cache, or in a specific area, and there is no specific limitation. In this way, each service node has the meta information of all data blocks in the distributed system.

[0133] In some implementations, the metadata of the data blocks stored by all service nodes in the distributed system may be registered in the external communication service process of each service node, so that each service node may subsequently search for the corresponding target metadata from the external communication service process running on each service node. For example, the metadata of the data block is registered in the external communication service process running on each service node. Taking the local service node as an example, the "obtaining the target metadata associated with the target data block identifier" in step 102 may include: searching for the corresponding target metadata from the local external communication service process of the local node according to the target data block identifier.

[0134] Specifically, in order to find the target data block corresponding to the target data block identifier from the local service node, it is necessary to first find the target meta information associated with the target data block identifier from the local service node, wherein, when the meta information of all data blocks is registered in the external communication service process respectively run by the service node, the meta information can be found from the external communication service process, therefore, taking the local service node as an example, the target meta information associated with the target data block identifier can be matched from the local external communication service process run by the local service node, so as to subsequently find the corresponding target data block from the storage of the local service node according to the target meta information. In this way, the desired target data block can be accurately found, and the search efficiency of the data block can be improved.

[0135] It should be noted that the primary purpose of registering data block metadata in the external communication service process is to ensure the accurate positioning of data blocks in the entire big data cluster (distributed system). Big data processing involves storing massive data blocks on many service nodes. The registration of data block metadata enables the external communication service process running on each service node to hold a consistent "data block storage map". Through this "data block storage map", the physical storage location (the address of the service node where it is located and / or the storage medium address on the service node) of the data block can be quickly determined based on the unique identifier of the data block, so as to achieve fast and accurate search and access of data blocks. For example, in large-scale data analysis tasks with distributed storage, data blocks may be stored in thousands of service nodes. The registered metadata is like index information. No matter which service node the data block is located on, it can be quickly located based on the metadata, laying the foundation for data processing and transmission, avoiding the inability of business services to run due to positioning errors or delays, and having reliability.

[0136] Therefore, at any time before searching for meta information, the meta information of each data block can be pre-registered in the external communication service process running on each service node. In addition, the meta information registered in the external communication service process running on each service node can also be updated at any time before or after this.

[0137] In some implementations, each service node can obtain the metadata of the data blocks stored on each service node in the distributed system, and register the metadata of each data block obtained in the external communication service process, so that the metadata of the data block is respectively registered in the external communication service process running on each service node. For example, before the step of "searching for the corresponding target metadata from the local external communication service process of the local node according to the target data block identifier", it can also include: obtaining the metadata of the data blocks pre-stored for the target business service by each service node in the distributed system, the metadata including the data block identifier and the storage node address; registering each metadata in the local external communication service process so that the local external communication service process records the storage node address associated with each data block identifier.

[0138] It should be noted that when the separation of computing and communication is achieved by running external communication service processes on each service node in the distributed system, in order to enable a service node to accurately request and obtain the target data block from another service node, the corresponding metadata of all data blocks stored on all service nodes in the distributed system is registered in the external communication service process running on each service node.

[0139] Data block metadata registration helps to ensure data consistency and communication reliability, where the registration process is initiated in all big data cluster nodes to ensure that the data block address space of the external communication service process of each node is consistent. This means that during data transmission and processing, each node has a unified understanding of the data block location to avoid data reading errors or communication confusion due to inconsistent information. Specifically, each service node can obtain the metadata of the data blocks stored on each node in the distributed system. The metadata can be generated by splicing the identifier of the corresponding data block and the storage node address (or the storage medium address on the service node), and broadcast and notify the data block metadata on each node between the service nodes in the distributed system, so that each service node in the distributed system can obtain the metadata of each data block in the system. Taking the local service node as an example, then, according to the metadata of all data blocks obtained, register in the external communication service process running on the local service node. In this way, the external communication service process running on each service node has the metadata of all data blocks in the distributed system, so that the external communication service process running on each service node can record the storage node address of each data block.

[0140] After that, a data block acquisition request is sent directly to the service node where the data block is located. In this way, when data blocks are transmitted between nodes, the sender and the receiver can accurately determine the source and destination of the data block based on the registered metadata, ensuring the complete and reliable transmission of data. In the scenario of high-concurrency data transmission or node failure recovery in the remote direct memory access network, unified metadata can ensure the accurate reconstruction of the communication link and the correct transmission of data, enhance the overall stability and fault tolerance of the system, and realize reliable processing of big data and efficient communication.

[0141] In some implementations, the corresponding target data block may be searched according to the target storage node address in the target meta information. For example, the step of "searching the target data block associated with the target meta information from the local storage" in step 102 may include:

[0142] (102.1) Determine the target storage node address corresponding to the target data block identifier according to the target meta information;

[0143] (102.2) When it is identified that the target storage node address is consistent with the node address of the local node, the target data block corresponding to the target data block identifier is searched from the local storage.

[0144] Among them, the target storage node address refers to the data block storage address indicated by the target meta information for the target data block identifier, that is, the target meta information contains the corresponding target data block identifier and the target storage node address. There is an association between the target data block identifier and the target storage node address. Through the target meta information, the target storage node address where the corresponding target data block identifier is located can be represented.

[0145] Specifically, when searching for a target data block, the service node needs to first determine whether the target storage node address indicated by the metadata registered for the data block identifier locally is consistent with the local node address. For example, taking the local service node as an example, after the local service node finds the target metadata associated with the target data block identifier, it can determine the target storage node address corresponding to the target data block identifier based on the target metadata. On the one hand, if the target storage node address is consistent with the node address of the local service node, it means that the target storage node address indicated by the target metadata for the target data block identifier is correct, and the target data block can be found from the local service node. Therefore, the local service node can accurately find the corresponding target data block from the local storage based on the target metadata.

[0146] On the other hand, when the target storage node address indicated by the target meta information is compared with the storage address of the local node, if the two are inconsistent, it means that the data block acquisition process is abnormal, and the abnormal information is returned to the data block requester (i.e., the target node). For example, it can also include: when it is identified that the target storage node address is inconsistent with the node address of the local node, the current time information is obtained, and the data block abnormal information is generated according to the current time information and the node address of the local node; the data block abnormal information is fed back to the target node.

[0147] The current time information may be real-time time information, such as “xx year xx month xx day xx hour xx minute xx second”.

[0148] Among them, the data block exception information can be an exception prompt information indicating that the data block cannot be obtained from the local node. Specifically, if it is detected that the target storage node address indicated by the target meta-information is inconsistent with the actual node address of the local service node, it means that the target data block corresponding to the target data block identifier is not stored in the local node, or the registered meta-information is wrong information, or the meta-information cannot be updated in time. Therefore, the local service node cannot provide the data block requester with direct access to the target data block. At this time, the current time information can be obtained. The current time information can be the time node information when the target storage node address is identified to be inconsistent with the node address of the local node, and the time node information and the node address of the local node are combined to generate data block exception information. In this way, the system is prevented from being attacked maliciously and intentionally sending non-existent data blocks. If the remote node is loaded without confirmation, a failure will occur.

[0149] Afterwards, the data block exception information is fed back to the target node to notify the target node so that other remedial measures can be taken, such as re-determining the storage location of the target data block to send a data block acquisition request, or feeding back the exception to the operator of the business service, etc.

[0150] In some embodiments, when the target storage node address indicated by the target meta information is compared with the storage address of the local node, if the two are inconsistent, it means that the local service node does not have the target data block corresponding to the target meta information, and the target data block can be searched in the distributed system, and the target data block is stored in the local service node to find the target data block. For example, it can also include: when it is identified that the target storage node address is inconsistent with the node address of the local node, download the target data block corresponding to the target data block identifier, and store the target data block in the local storage, so that the local storage has the target data block corresponding to the target meta information.

[0151] The downloading of the target data block may specifically be downloading the target data block corresponding to the target data block identifier from the service end of the business service, or searching for the target data block corresponding to the target data block identifier from each service node of the distributed system, which is not limited here. In this way, the target data block is stored on the local service node, so that the storage location of the target data block matches the actual target meta information in the external communication service process.

[0152] In addition, when it is identified that the target storage node address is inconsistent with the node address of the local node, the local service node can also forward the current data block acquisition request to each service node in the distributed system, and when receiving the target data block storage address fed back by other target service nodes, the target metadata in the local external communication service process running by the local service node is updated according to the target data block storage address. Specifically, the target storage node address associated with the target data block identifier in the target metadata is updated to obtain the updated target metadata. Then, the target data block sent by other target service nodes can be received so that the target data block can be subsequently loaded into the target communication queue pair for the target node and wait for rapid transmission to the target node. A data block delegation sending request can also be sent to other target service nodes to entrust other target service nodes to send the target data block to the target node through the corresponding communication queue pair.

[0153] Through the above method, the corresponding target metadata can be searched in the local service node according to the target data block identifier carried in the data block acquisition request, and the corresponding target data block can be searched according to the target metadata. In this way, when computing and communication are separated, preliminary preparations can be made for the subsequent delivery of target data blocks, thereby improving the transmission efficiency of data blocks.

[0154] 103. Determine a target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair.

[0155] In an embodiment of the present application, after finding the target data block corresponding to the target metadata, the target data block needs to be loaded into a specific target communication queue pair. Specifically, since each service node in the distributed system pre-creates a communication queue pair for each other service node, that is, it contains multiple communication queue pairs, it is necessary to find a target communication queue pair that the local service node uses specifically for communication with the target node, so as to load the target data block into the target communication queue pair to wait for transmission, thereby preparing for the rapid delivery of subsequent target data blocks and improving the transmission efficiency of data blocks.

[0156] Among them, the target communication queue pair refers to a queue pair that the local service node uses specifically to communicate with the target node, which can be composed of a target receiving queue and a target sending queue. The target receiving queue is used to receive any information such as data blocks and requests sent by the target node, and the target sending queue is used to temporarily store any information such as data blocks and requests sent to the target node. In this way, the queues are divided according to the data transmission direction, and the reception and transmission of data are managed separately, so that data reception and transmission between service nodes can be carried out independently, and data transmission and reception can be carried out at the same time, avoiding data transmission running the progress of data reception, or avoiding data reception affecting the progress of data transmission, so that the timeliness requirements of data transmission and data reception can be met between the local service node and the target node.

[0157] In some embodiments, the local external communication service process run by the local service node includes multiple communication queue pairs, each communication queue pair is bound to a service node identifier, the service node identifier includes but is not limited to a node number, a node address, etc., indicating that the communication queue pair is dedicated to the communication between the local service node and the service node corresponding to the service node identifier. To determine the target communication queue pair pre-created by the local node for the target node, the target node identifier corresponding to the target node can be determined, and the target communication queue pair pre-created by the local node for the target node can be determined from the local external communication service according to the target node identifier.

[0158] In some implementations, data blocks are generally stored in the storage of the service node. Before transmission is required, the data blocks need to be loaded into a cache that can be read quickly to wait for the fast transmission of the data blocks. For example, "loading the target data block into the target communication queue pair" in step 103 may include:

[0159] (103.1) transferring the target data block in the local storage to the main storage space of the local node, where the main storage space is used to store data that is directly read and written by the local processor;

[0160] (103.2) Copy the target data block in the main storage space to the target communication queue pair.

[0161] Specifically, the target data block can be stored in the local storage space of the local service node. The local storage (space) can be a local disk. The read and write speed of the local disk is slow, while the local main memory has the characteristics of fast access rate and is a storage space that can be directly addressed by the central processing unit. In order to improve the data transmission rate, the local service node can read the target data block from the local disk to the local main memory through the running resources specifically used for the communication process. Further, the target data block in the local main memory is copied to the target communication queue pair. Specifically, the target data block can be copied to the target cache corresponding to the target sending queue of the target communication queue pair to wait for the target data block in the target communication queue pair to be sent to the target node through the remote direct memory access network. In this way, the separation of computing and communication is realized. The local service node processes the communication process through a specifically allocated resource pool. The target node of the computing node role does not need to consume a large amount of resources in processing the communication process, so that the running resources of the target node are focused on computing, improving data computing efficiency and ensuring the normal operation of business services.

[0162] In some implementations, the sending and receiving of data blocks are managed separately through different queues. When a data block needs to be sent, the target data block needs to be transmitted through the sending queue. For example, the target communication queue pair includes a target sending queue and a target receiving queue, and step (103.2) may include: copying the target data block in the main storage space to the target sending queue in the target communication queue pair.

[0163] Specifically, for the target communication queue pair between the local service node and the target node, including the target sending queue for the target node and the target receiving queue for the target node, after loading the target data block in the local storage into the main storage space, the local service node can copy the target data block in the main storage space to the target sending queue in the target communication queue pair, and specifically copy it to the target cache of the target sending queue, waiting for the target data block in the target communication queue pair to be sent to a corresponding communication queue pair of the target node through the remote direct memory access network. In this way, the data between receiving and sending are independent of each other, avoiding data interference, so as to facilitate the resource allocation and management of threads, such as allocating more thread resources as backup for the sending queue, ensuring the transmission reliability of data blocks, and improving the transmission speed of data blocks.

[0164] Through the above method, the local external communication service process of the local service node includes multiple communication queue pairs, and the target communication queue pair specifically used by the local service node for communication with the target node can be determined, so that the target data block can be loaded into the target communication queue pair to wait for transmission. In this way, preparations can be made for the rapid delivery of subsequent target data blocks, thereby improving the transmission efficiency of data blocks.

[0165] 104. Transmit the target data block in the target communication queue pair to the target node through the remote direct memory access network.

[0166] In an embodiment of the present application, after the target data block is loaded into a target communication queue pair dedicated to communication with a target node, the target data block can be directly copied to the target communication queue pair in advance through the communication queue pair mechanism of the remote direct memory access network. This process does not require the intervention of the central processor of the local service node to quickly transmit the target data block in the target communication queue pair to the target node. In this way, direct transmission of data blocks is achieved to improve data transmission efficiency. In this way, the data acquisition speed of the business service can still be met under the separation of computing and communication, and the normal calculation of the data of the business service can be maintained, so that the business service can run normally and stably with reliability.

[0167] Among them, the remote direct memory access network allows the local service node to directly access the memory, cache, etc. of the target node through the remote direct memory access network, or allows the target node to directly access the memory, cache, etc. of the local service node through the remote direct memory access network without the intervention of the central processor on the local service node or the target service node, thereby realizing high-throughput, low-latency data block transmission.

[0168] In some implementations, the local service node transmits the target data block in the target communication queue pair by starting the target function corresponding to the remote direct memory access technology. For example, step 104 may include:

[0169] (104.1) running a direct transfer function corresponding to a remote direct memory access network;

[0170] (104.2) The target data block in the target communication queue pair is transmitted to the target node through the direct transmission function.

[0171] Among them, the direct transmission function can be a source function related to the remote direct memory access network, which can be understood as an operation function for data block transmission. By running the direct transmission function, the target data block can be transmitted to the target node. Specifically, the local service node obtains the direct transmission function from the relevant library of the remote direct memory access network, and then obtains the first queue identifier of the target sending queue in the target communication queue pair, and obtains the second queue identifier of the receiving queue in the communication queue pair of the target node for the local service node. The first queue identifier, the target data block identifier and the second queue identifier are used as function parameters to execute the direct transmission function to send the target data block in the target communication queue pair to the receiving queue corresponding to the second queue identifier of the target node, so as to realize the rapid transmission of the target data block. In this way, the data transmission efficiency is improved, the data calculation of the business service is maintained normally, and the business service can run normally and stably.

[0172] In the embodiment of the present application, although the remote direct memory access network satisfies the fast transmission of data blocks between any two service nodes in the distributed system, when a communication failure occurs in the remote direct memory access network, the data blocks cannot be transmitted normally, affecting the transmission progress of the data blocks. Therefore, in order to achieve reliable transmission, it is necessary to detect the status of the remote direct memory access network in real time or periodically, and timely discover the communication capability of the remote direct memory access network, so as to take corresponding response strategies in the event of a communication failure to improve the reliability of data block transmission.

[0173] In some embodiments, before transmitting the target data block in the target communication queue pair, the communication capability of the remote direct memory access network may be evaluated based on the transmission delay duration and packet loss rate of the remote direct memory access network to decide whether to transmit the target data block through the remote direct memory access network. For example, before step 104, the following may be included: obtaining the current transmission delay duration and packet loss rate of the remote direct memory access network. Then step 104 may include: when the transmission delay duration is less than a preset delay threshold, and the packet loss rate is less than a preset packet loss rate threshold, transmitting the target data block in the target communication queue pair to the target node through the remote direct memory access network.

[0174] The transmission delay duration may be the waiting time for the local service node to transmit the data block to the corresponding service node through the remote direct memory access network during the last data block transmission in the historical time, or the time spent by the remote direct memory access network for the last data block transmission.

[0175] Among them, the packet loss rate can be the ratio of the number of packet losses when the local service node transmits data blocks through the remote direct memory access network. Specifically, the total amount of data block transmission for a specific time and the number of data block transmission failures can be obtained, and the packet loss rate can be determined based on the ratio between the number of data block transmission failures and the total amount of data block transmission.

[0176] Among them, the preset delay threshold can be a determination value for determining the size of the data block transmission delay. It can be set according to an empirical value or according to the normal transmission time of a conventional transmission protocol. For example, it can be set according to the transmission control protocol (TCP) for the transmission time of a data block. For example, the transmission time for transmitting a data block based on the transmission control protocol is used as the preset delay threshold. In this way, when the time spent by the remote direct memory access network in transmitting a data block is less than the preset delay threshold, it indicates that there is a transmission delay.

[0177] Specifically, before transmitting the target data block in the target communication queue pair through the remote direct memory access network, the transmission delay time and packet loss rate from the remote direct memory access network to the current state can be obtained first, and the transmission delay time can be compared with a preset delay threshold, which can be set according to an empirical value, and the packet loss rate can be compared with a preset packet loss rate threshold. When the transmission delay time is less than the preset delay threshold and the packet loss rate is less than the preset packet loss rate threshold, it indicates that the communication capability of the remote direct memory access network is normal and there is no communication failure. Therefore, the target data block in the target cache of the target sending queue of the target communication queue pair can be transmitted to the target node through the remote direct memory access network, so that the target node obtains the target data block from the receiving queue of the communication queue pair of the local service node, and performs data calculation, so that the business service can run normally.

[0178] In some embodiments, when transmission delay or serious packet loss occurs, the data transmission mode can be switched to transmit the target data block through other data communication channels. For example, when transmission delay or serious packet loss occurs, it can also include: when the transmission delay time is greater than or equal to the preset delay threshold, or the packet loss rate is greater than or equal to the preset packet loss rate threshold, obtaining the network address of the target node; establishing a data transmission channel between the local node and the target node of the network address through the socket interface; when the target data block is queried from the main storage space of the local node, transmitting the target data block in the main storage space to the target node corresponding to the network address through the data transmission channel; when the target data block is not queried from the main storage space, loading the target data block from the local storage into the main storage space, and transmitting the target data block in the main storage space to the target node corresponding to the network address through the data transmission channel.

[0179] Specifically, before transmitting the target data block in the target communication queue pair through the remote direct memory access network, if it is detected that the transmission delay time is greater than or equal to the preset delay threshold, or the packet loss rate is greater than or equal to the preset packet loss rate threshold, it means that the remote direct memory access network has abnormal communication capability and communication failure. At this time, it is necessary to switch to other data communication channels to transmit the target data block, for example, using the transmission control protocol (TCP) to transmit the target data block. Specifically, the network address of the target node is obtained; through the communication socket interface, a data transmission channel is established between the local service node and the target node corresponding to the network address. The data transmission channel refers to a communication channel based on the transmission control protocol (TCP) for the subsequent transmission of the target data block.

[0180] Furthermore, due to the communication failure of the remote direct memory access network, the local service node cannot use the remote direct memory access network to transmit the target data block at this time, then the target data block previously loaded into the target communication queue pair may be lost, and the target data block in the target communication queue pair only supports the remote direct memory access network for transmission. At this time, it is necessary to reacquire the target data block. Specifically, the local service node can first search for the target data block corresponding to the target data block identifier from the main storage space, and the main storage space belongs to the storage area that the central processor can directly operate. By first searching for the target data block corresponding to the target data block identifier from the main storage space, the acquisition efficiency of the target data block can be improved. On the one hand, if the target data block corresponding to the target data block identifier is found from the main storage space, the target data block is sent to the target node through the data transmission channel corresponding to the transmission control protocol; on the other hand, if the target data block corresponding to the target data block identifier cannot be found from the main storage space, it is necessary to reload the target data block from the local storage to the main storage space, and then send the target data block in the main storage space to the target node through the data transmission channel corresponding to the transmission control protocol. In this way, when a communication failure occurs in the remote direct memory access network, another data transmission channel can be selected to transmit the target data block, ensuring that the target data block can be transmitted to the target node, thereby improving data transmission reliability.

[0181] In the embodiment of the present application, any service node in the distributed system can act as a data block requester to obtain data blocks from other service nodes for data calculation to perform business services, and can also act as a data block remote provider to transmit the target data block to the service node of the data block requester. Based on this, taking the local service node as an example, in addition to responding to the data block acquisition request of the target node and transmitting the target data block to the target node, the local service node can also act as a data block requester to request other service nodes to provide data blocks for calculation, so as to meet the business service's needs for performing big data processing in a distributed system, so that the business service can run normally.

[0182] In some embodiments, the local service node may determine the required business data block, determine the storage node address of the business data block according to the metadata of the business data block, and request to obtain the business data block through a specific communication queue to the storage node address where the business data block is located. For example, it may also include: determining the business data block identifier obtained by the local node for the target business service request; querying the remote node address corresponding to the business data block identifier based on the local external communication service process; sending a business data block acquisition request to the target service node corresponding to the remote node address in the distributed system, the data block acquisition request carrying the business data block identifier; obtaining the business data block corresponding to the business data block identifier fed back by the target service node through the target receiving queue in the communication queue pair pre-created between the local node and the target service node, so as to run the target business service according to the business data block.

[0183] The business data block refers to the data block required by the business service during operation, such as the material data required to render an advertisement page, etc. The description of the business data block can refer to the description of the "data block" and "target data block" above, which will not be repeated here. Similarly, the business data block identifier can refer to the description of the data block identifier, which will not be repeated here.

[0184] Specifically, first, the local service node can determine the required business data block for the running task of the target business service, and determine the business data block identifier corresponding to the business data block, and then query the business data block meta information associated with the business data block identifier from the local external communication service process running on the local service node, and then determine the remote node address of the business data block according to the business data block meta information, and the remote node address is the storage node address, which can refer to the description of the aforementioned "storage node address". Then, a business data block acquisition request is generated according to the business data block identifier, and the business data block acquisition request is sent to the target service node corresponding to the remote node address, so that the target service node loads the business data block into the sending queue of the communication queue pair of the target service node for the local service node according to the aforementioned "target data block" search and loading process, and sends the business data block in the sending queue to the target receiving queue of the communication queue pair pre-created between the local service node and the target service node through the remote direct memory access network, so that the business data block fed back by the target service node can be obtained, and then the target business service is run according to the business data block. In this way, the needs of business services to perform big data processing in a distributed manner are met, allowing business services to run normally.

[0185] Through the above method, the target data block in the target communication queue pair can be transmitted to the target node through the remote direct memory access network to improve the data transmission efficiency. In this way, the data acquisition speed of the business service can still be met under the separation of computing and communication, and the data calculation of the business service can be maintained normally, so that the business service can run normally and stably with reliability.

[0186] From the above overall description of the embodiment of the present application, it can be seen that the embodiment of the present application forms a distributed system corresponding to the big data processing framework through multiple service nodes, and each service node is registered with the meta information of each data block stored on all service nodes. According to the meta information of each data block, the storage node address corresponding to each data block in the distributed system can be quickly located, so that there is no need to traverse and send data block acquisition requests to all service nodes in the distributed system, reduce network interaction, and improve data acquisition efficiency. In addition, based on the remote direct memory access network, the data block can be remotely transmitted to other service nodes, thereby realizing the separation of data calculation and communication, avoiding the simultaneous operation of the calculation process and the communication process on a single service node, which makes the operation load of the service node too high, thereby avoiding the risk of data transmission failure and improving the reliability of data transmission. At the same time, for any two service nodes in the distributed system of the big data processing framework, a communication queue pair is established separately, and the communication queue pair can be reused for multiple data block transmissions between the two service nodes, avoiding frequent full connections between any two service nodes. In this way, while satisfying reliable transmission in big data processing, communication delays are further avoided, thereby improving data block transmission efficiency. In addition, when separating data calculation and communication, the computing resources of the currently used service nodes are utilized, and each service node can be used as both a computing end and a communication end, so there is no need to set up an additional communication server, reducing equipment costs.

[0187] In combination with the method described in the above embodiment, the method will be further described in detail with examples below.

[0188] Figure 3 This is another step flow chart of the data transmission method provided in the embodiment of the present application. Figure 3 Give a description.

[0189] In the embodiment of the present application, the data transmission method is executed by a computer device, such as Figure 1 The specific process of the data transmission method is as follows:

[0190] 201. Run the remote task program on the local node, and start the local external communication service process on the local node after running the remote task program.

[0191] In an embodiment of the present application, a business service can be jointly run based on multiple service nodes in a distributed system, wherein data blocks related to the business service are distributedly stored in the distributed system, and the business service is run while separating the computing process and the communication process for the business service. Specifically, in order to separate the computing process and the communication process, first, the same remote task program is run on each service node respectively, so that the multiple service nodes running the remote task program together constitute a distributed system corresponding to the big data processing framework. Then, an external communication service process needs to be run on each service node in the distributed system, which is the basis for the subsequent separation of the computing process and the communication process on the physical service node, effectively reducing the computing process workload, avoiding the contention of computing resources for communication operations, allowing the computing process to focus on the core tasks of data processing, improving data processing efficiency, and improving the response speed of business services.

[0192] Among them, the remote task program can be understood as a program of the big data processing framework of the business service, which is a tool for processing large-scale data in a distributed system. The role of the remote task program includes at least three stages of data processing in the business service: in the data block registration stage, the remote task program is responsible for marking the data block global unique identifier (i.e., data block identifier), ensuring the consistency and uniqueness of the data block meta-information, and synchronizing it to each service node in the distributed system, laying the foundation for subsequent data positioning and transmission. In the data block acquisition request initiation stage, the remote task program on the target node generates a request instruction based on the task requirements, triggers the communication process, and coordinates the data transmission operation between the target node and the local node, including allocating worker threads, managing thread communication and data transmission status, etc. In the data processing stage, the remote task program on the target node organizes computing node resources, schedules and executes data processing tasks (such as data analysis and mining algorithms), and collaborates with the communication framework to achieve efficient coordination of computing and communication, ensuring the smooth flow and processing of data between each service node, and is a key supporting component for realizing reliable processing and efficient communication of big data.

[0193] In addition, it is also necessary to run an external communication service process on each service node. Specifically, in a distributed system, the external communication service process, as an independent functional module, focuses on network communication-related operations, such as data transmission, connection management, communication protocol processing, etc., to ensure that data is accurately and efficiently exchanged between nodes, and cooperates with the computing process to improve the overall performance and reliability of the distributed system. It should be noted that the external communication service process is the key to realizing the separation of computing and communication architecture. By running independently on a service node in a distributed system (such as the "local node" from the perspective of the embodiment of the present application), the workload of the computing process in the service node is effectively reduced, and the communication operation is avoided. The computing resource contention on the service node allows the service node computing process to focus on the core task of data processing and improve computing efficiency. The function of pre-establishing a communication queue pair is to accelerate data transmission, reduce communication delays, and improve the overall communication performance of the system. It works closely with the program of the big data processing framework to realize functions such as data block meta-information registration management, data block acquisition request response, and data transmission control, ensuring that the communication link in the big data processing process is reliable and efficient, and is an important part of ensuring the high reliability and high performance of the big data processing system, and provides strong support for the stable operation of big data applications in complex network environments.

[0194] 202. Create communication queue pairs with other service nodes in the distributed system through the local external communication service process.

[0195] In the embodiment of the present application, in the external communication process running on each service node, a communication queue pair must be created separately for any other service node in the distributed system. In this way, the external communication service process running on each service node contains multiple communication queue pairs for communicating with different other service nodes. In this way, the function of pre-establishing the communication queue pair is to accelerate data transmission, reduce communication delays, and improve the overall communication performance of the system. It works closely with the big data processing framework program to ensure that the communication link in the big data processing process is reliable and efficient. It is an important part of ensuring the high reliability and high performance of the big data processing system, and provides strong support for the stable operation of big data applications in complex network environments.

[0196] Specifically, in the case of separating computing and communication, in order to directly and quickly obtain data blocks from remote service nodes, the remote direct memory access (RDMA) network is used to quickly transmit data blocks to ensure the normal operation of business services. Based on this, each service node can start the queue pair creation process through the relevant configuration acquisition function in the interface library of the remote direct memory access network. Taking the local service node as an example, the local service node obtains the local communication configuration parameters through this function, so as to allocate corresponding memory resources for the communication process in combination with the local communication configuration parameters, which can be specifically running memory resources. The running memory resources allocated by the local service node for communication are mainly used to run the local external communication service process to meet the purpose of remote and rapid transmission of data blocks. Therefore, the local service node can create a communication queue pair for each other service node in the distributed system in the external communication service process running in the local service node based on the allocated running memory resources, so as to reduce the communication delay of the data block, improve the overall communication performance of the system, and promote the stable operation of business services.

[0197] 203. Obtain the meta information of the data blocks pre-stored for the target business service in each service node in the distributed system, and register each meta information in the local external communication service process.

[0198] In an embodiment of the present application, at any time before searching for meta-information, the meta-information of each data block can be pre-registered in the external communication service process running on each service node. In addition, the meta-information registered in the external communication service process running on each service node can also be updated at any time before or after this. The primary purpose of registering data block meta-information in the external communication service process is to ensure the accurate positioning of the data block in the entire big data cluster (distributed system). Big data processing involves the storage of massive data blocks on many service nodes, and the registration of meta-information of data blocks enables the external communication service process running on each service node to hold a consistent "data block storage map". Through this "data block storage map", the physical storage location (the address of the service node and / or the storage medium address on the service node) of the data block can be quickly determined based on the unique identifier of the data block, so as to realize fast and accurate search and access of data blocks.

[0199] Specifically, the data block meta-information registration process is initiated on all big data cluster nodes to ensure that the data block address space of the external communication service process of each node is consistent. This means that during data transmission and processing, each node has a unified understanding of the data block location, avoiding data reading errors or communication confusion due to inconsistent information. Specifically, each service node can obtain the meta-information of the data blocks stored on each node in the distributed system. The meta-information can be generated by splicing the identifier of the corresponding data block and the storage node address (or the storage medium address on the service node), and broadcast and notify the data block meta-information on each node between the service nodes in the distributed system, so that each service node in the distributed system can obtain the meta-information of each data block in the system. Taking the local service node as an example, further, according to the meta-information of all data blocks obtained, it is registered in the external communication service process running on the local service node. In this way, the external communication service process running on each service node has the meta-information of all data blocks in the distributed system, so that the external communication service process running on each service node can record the storage node address of each data block.

[0200] 204. Obtain a data block request sent by the target node.

[0201] In the embodiment of the present application, the target node, as the computing end of the business service, sends a data block acquisition request to the local node (a service node other than the target node in the distributed system), and the data block acquisition request carries the target data block identifier, hoping to obtain the target data block required by the target data block identifier from the local node. In this way, the communication of data blocks is realized while the computing process and the communication process are separated independently, thereby improving the system performance, thereby improving the reliability of data transmission, ensuring that the required data blocks can be provided for the operation of the business items of the business service, and ensuring the stability of the business service.

[0202] 205. Obtain target meta information associated with the target data block identifier, and search for the target data block associated with the target meta information from the local storage.

[0203] In an embodiment of the present application, after receiving a data block acquisition request sent by any target node in the distributed system, the local service node can search for the corresponding target metadata in the local service node according to the target data block identifier carried in the data block acquisition request, and thus search for the corresponding target data block according to the target metadata. In this way, when computing and communication are separated, preliminary preparations can be made for the subsequent delivery of target data blocks, thereby improving the transmission efficiency of data blocks.

[0204] Specifically, in order to find the target data block corresponding to the target data block identifier from the local service node, it is necessary to first find the target meta information associated with the target data block identifier from the local service node, wherein, when the meta information of all data blocks is registered in the external communication service process respectively run by the service node, the meta information can be found from the external communication service process, therefore, taking the local service node as an example, the target meta information associated with the target data block identifier can be matched from the local external communication service process run by the local service node, so as to subsequently find the corresponding target data block from the storage of the local service node according to the target meta information. In this way, the desired target data block can be accurately found, and the search efficiency of the data block can be improved.

[0205] Furthermore, when searching for the target data block, the service node needs to first determine whether the target storage node address indicated by the metadata registered for the data block identifier locally is consistent with the local node address. For example, taking the local service node as an example, after the local service node finds the target metadata associated with the target data block identifier, it can determine the target storage node address corresponding to the target data block identifier based on the target metadata. On the one hand, if the target storage node address is consistent with the node address of the local service node, it means that the target storage node address indicated by the target metadata for the target data block identifier is correct, and the target data block can be found from the local service node. Therefore, the local service node can accurately find the corresponding target data block from the local storage based on the target metadata.

[0206] 206. Determine a target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair.

[0207] In an embodiment of the present application, after finding the target data block corresponding to the target metadata, the target data block needs to be loaded into a specific target communication queue pair. Specifically, since each service node in the distributed system pre-creates a communication queue pair for each other service node, that is, it contains multiple communication queue pairs, it is necessary to find a target communication queue pair that the local service node uses specifically for communication with the target node, so as to load the target data block into the target communication queue pair to wait for transmission, thereby preparing for the rapid delivery of subsequent target data blocks and improving the transmission efficiency of data blocks.

[0208] Specifically, the local external communication service process run by the local service node includes multiple communication queue pairs, each communication queue pair is bound to a service node identifier, the service node identifier includes but is not limited to a node number, a node address, etc., indicating that the communication queue pair is dedicated to the communication between the local service node and the service node corresponding to the service node identifier. To determine the target communication queue pair pre-created by the local node for the target node, the target node identifier corresponding to the target node can be determined, and the target communication queue pair pre-created by the local node for the target node can be determined from the local external communication service according to the target node identifier.

[0209] Furthermore, the target data block can be stored in the local storage space of the local service node, which can be a local disk. The read and write speed of the local disk is slow, while the local main memory has the characteristics of fast access rate and is a storage space that can be directly addressed by the central processor. In order to improve the data transmission rate, the local service node can read the target data block from the local disk to the local main memory through the running resources specifically used for the communication process. Further, the target data block in the local main memory is copied to the target communication queue pair, which can be specifically copied to the target cache corresponding to the target sending queue of the target communication queue pair, waiting for the target data block in the target communication queue pair to be sent to the target node through the remote direct memory access network. In this way, the separation of computing and communication is realized, and the local service node processes the communication process through a specifically allocated resource pool. The target node of the computing node role does not need to consume a large amount of resources in processing the communication process, so that the running resources of the target node are focused on computing, improving data computing efficiency and ensuring the normal operation of business services.

[0210] 207. Transmit the target data block in the target communication queue pair to the target node through the remote direct memory access network.

[0211] In an embodiment of the present application, after the target data block is loaded into a target communication queue pair dedicated to communication with a target node, the target data block can be directly copied to the target communication queue pair in advance through the communication queue pair mechanism of the remote direct memory access network. This process does not require the intervention of the central processor of the local service node to quickly transmit the target data block in the target communication queue pair to the target node. In this way, direct transmission of data blocks is achieved to improve data transmission efficiency. In this way, the data acquisition speed of the business service can still be met under the separation of computing and communication, and the normal calculation of the data of the business service can be maintained, so that the business service can run normally and stably with reliability.

[0212] Specifically, the local service node obtains the direct transmission function from the relevant library of the remote direct memory access network, and then obtains the first queue identifier of the target sending queue in the target communication queue pair, and obtains the second queue identifier of the receiving queue in the communication queue pair of the target node for the local service node, and executes the direct transmission function with the first queue identifier, the target data block identifier and the second queue identifier as function parameters to send the target data block in the target sending queue of the target communication queue pair to the receiving queue corresponding to the second queue identifier of the target node, so as to realize the rapid transmission of the target data block. In this way, the data transmission efficiency is improved, the data calculation of the business service is maintained normally, and the business service can run normally and stably.

[0213] In order to facilitate the understanding of the embodiment of the present application, the embodiment of the present application will be described with a specific application scenario example. Specifically, the application scenario example is described by executing the above steps 201-207.

[0214] It should be noted that the data transmission method is suitable for the big data computing scenario of application services, which can be understood as batch data computing, including two parts: computing and communication. The computing process and the communication process are closely integrated and interdependent. For example, when executing computing tasks, data communication is required frequently to obtain necessary information or transmit intermediate computing results, and the results of communication directly affect the subsequent computing. In order to improve the transmission reliability and efficiency of data, the big data processing framework program is run separately through multiple service nodes to form a distributed system corresponding to the big data processing framework based on these multiple service nodes, and each service node runs an external communication service process separately. The external communication service process of each node is registered with the metadata of the data blocks stored on all service nodes. Subsequently, based on the remote direct memory access network, the data blocks on other service nodes can be remotely read to separate the data computing and communication, avoid the risk of transmission failure caused by excessive blocking of threads on a single node, and improve the reliability of data transmission. At the same time, a communication queue pair is established separately between any two service nodes in the distributed system of the big data processing framework. The communication queue pair can be reused to meet the multiple data block transmissions between the two service nodes, avoiding frequent connection establishment between the two service nodes. In this way, reliable transmission is met in big data processing while improving the data block transmission efficiency. The following introduces the data transmission method using the data transmission scenario example.

[0215] 1. The data transmission scenario example is briefly introduced as follows:

[0216] In large-scale RDMA networks for big data applications, business requests often fail due to a series of problems such as RDMA network failure during big data communication execution, excessive data reading and writing pressure, and excessive thread load. When the amount of data involved in the request is large, the application needs to use thousands of computing, storage servers, and network devices, and any single point of failure may cause the request to fail. The current big data processing system lacks fault tolerance in the face of request failures, and the business (application) service needs to be repeatedly restarted when data transmission errors occur. Therefore, ensuring the reliability of big data processing is conducive to avoiding the time and cost losses caused by request failures and achieving lower latency and accurate big data processing services.

[0217] Related technologies frequently establish connections between the storage side and the computing end to achieve full connection between the storage side and the computing end to achieve reliable data transmission. Some use dedicated communication servers to reliably transmit data. However, frequent connection establishment reduces data transmission efficiency, and the use of dedicated communication servers requires higher equipment costs.

[0218] To address the above issues, this data transmission scenario example provides a high-reliability RDMA network communication solution for large-scale big data services, involving the field of big data communication. The main process includes abstracting the communication of big data into an independent process, pre-establishing a communication channel, and using the Transmission Control Protocol (TCP) as an escape transmission method to achieve the reliable communication that big data deserves.

[0219] It should be noted that this data transmission scenario example can be applied to big data processing in application services of Internet companies, cloud service providers, etc. For example, for communication-intensive applications that mainly perform big data mapping or merging tasks, the execution efficiency of user requests is closely related to network quality. Once continuous packet loss or congestion occurs in the network, the RDMA network will be unstable for a short period of time, and user requests may face repeated failures. Alternatively, for computing-communication coupling applications that mainly perform sorting, in the current big data distributed computing system that mainly performs thread-level parallelism, threads often crash due to excessive computing and communication loads. Once a thread crashes, user requests either fail or return incorrect results. In response to the problems in the above two scenarios, this data transmission scenario example makes software-level optimization for RDMA network problems. Specifically, it optimizes the high-performance remote direct memory access network of the big data business cluster by completely separating the computing and communication in big data processing on the service node side, so that computing and communication are not on the same node, and using independent communication processes to handle all communication operations, thereby reducing the workload of the computing process and avoiding a series of problems caused by high thread load pressure, such as communication thread crash, reducing the risk of data transmission failure, and combining the Transmission Control Protocol (TCP) as an emergency transmission path for transmission failures to achieve reliable communication of big data and ensure the robustness of communication in big data business scenarios.

[0220] 2. The specific implementation process of this data transmission scenario example is as follows:

[0221] This data transmission scenario example can completely separate computing and communication on the service node side, and use TCP as an escape communication solution to ensure the reliability of big data communication. The main steps of this method include six contents: external communication service startup, data block registration, data block acquisition request, data block query, data block communication, and communication service termination. Figure 4 This is an example diagram of the architecture of the data transmission scenario provided in the embodiment of the present application, combined with Figure 4 As shown, the data transmission scenario is as follows:

[0222] (1) External communication service startup: When the application service is initialized, an independent external communication service process is started, and a communication channel (i.e., a communication queue pair) is pre-established for other nodes to speed up the subsequent communication process.

[0223] The initialization application process may include starting a big data processing framework program and a backend database on all service nodes of a big data node cluster (i.e., a distributed system). The present invention simultaneously starts an external communication service process when initializing the application, and after the process is started, a thread pool is maintained for subsequent communication execution.

[0224] In addition, a communication channel is pre-established in the external communication service process on each service node. Specifically, in the RDMA network, the communication between any two service nodes requires a separate queue pair to be established, and the communication queue pair between service nodes can be reused between different communications. By synchronously creating an RDMA communication queue pair between every two nodes when the external communication service is established, the delay of subsequent communications is saved.

[0225] (2) Data block registration: After detecting that the big data processing framework needs to involve data communication, the big data processing framework will register the metadata to be pulled with the established external communication service. This registration process will be initiated on all nodes in the big data cluster to ensure that all external communication service processes have the same data block address space.

[0226] Among them, after detecting that the big data processing framework initiates a communication request, it can be understood that the application service needs to involve data communication in the process of executing business services. At this time, the relevant business data that the application service needs to transmit / communicate can be pre-saved in the local storage of each service node in the distributed system in the form of a database.

[0227] Furthermore, the big data processing framework marks each data block with a globally unique identifier (i.e., data block identifier), and collectively refers to the data block identifier and the data block storage location (which can be understood as the storage service node address, such as the Internet Protocol address of the service node) as metadata, and registers the metadata with the external communication service process on each service node; the registration process will be initiated on all service nodes in all big data node clusters to ensure that the external communication service processes on all nodes have the storage information of all data blocks, so that they can subsequently determine the storage location of the data block in the big data node cluster based on the data block identifier.

[0228] (3) Data block acquisition request: When receiving a data block pull request initiated by the computing end of the application service (which can be understood as the upstream application service), the big data processing framework immediately initiates a remote transmission request to the node where the data block is located and confirms the validity of the reply. The remote node allocates the corresponding communication thread for the request and is responsible for the data block transmission.

[0229] It should be noted that the service nodes involved in the data communication process can be classified into two types, including remote nodes (which can be understood as the executing entity of the embodiment of the present application) and request nodes. The request node refers to the node that needs to pull the data block, and the remote node refers to the storage node of the data block that needs to be transmitted during the communication process.

[0230] Among them, when a data block pull request is generated at the requesting node, the big data processing framework program on the requesting node will assign a worker thread to be responsible for the transmission of the data block. Specifically, the worker thread will send a special data block acquisition request packet to the remote node, and the request data packet carries the unique identifier of the transmitted data block.

[0231] After receiving the data block acquisition request packet, the remote node immediately applies for a worker thread from the thread pool for communication and data transmission with the requesting node. The worker thread of the remote node will bind a pre-assigned communication queue pair, which has all the information required to communicate with the requesting node. The worker thread of the remote node will send a reply to the requesting node agreeing to pull the data to ensure normal data block transmission and reception.

[0232] (4) Data block query: When a data block acquisition request is sent to a remote node, the remote node’s external communication service quickly searches for the data block in the local storage based on the metadata and loads it into the memory buffer of the RDMA network card.

[0233] When the external communication service process of the remote node (which can be the service node corresponding to the execution subject of this data transmission scenario instance) receives a data block transmission request, the remote node needs to confirm whether the required target data block is stored locally based on the metadata in the local storage. If the target data block is not local, the worker thread of the remote node returns a data not found exception message through the queue pair between the remote node and the requesting node. If the target data block is in the local storage of the remote node, the corresponding data block is transferred from the local storage (local hard disk) to the main memory of the remote node, and the data block is copied to the registered memory of the queue pair (that is, the cache of the RDMA network card) in the main memory to wait for RDMA network transmission.

[0234] (5) Data block communication: The target data block transmission process is mainly based on the RDMA network. The data block transmission success rate is related to the RDMA network.

[0235] On the one hand, if the RDMA network connection is normal, the target data block in the buffer will be transmitted between the requesting node and the remote node based on the pre-established transmission queue to transmit the target data block to the requesting node. For example, if the RDMA communication capability is normal and there is no obvious packet loss, the remote node worker thread directly transmits the data through the RDMA source language and queue pair. The requesting node worker thread is responsible for receiving the data and storing it in the local storage of the requesting node.

[0236] On the other hand, if the RDMA network has a connection abnormality, for example, the RDMA communication capability is abnormal, including the inability to send data packets and the data packet sending timeout, a TCP transmission connection will be established between the requesting node and the remote service node based on the communication socket (SOCKET) interface. Specifically, the worker thread of the remote service node will establish a TCP protocol connection with the requesting node through the SOCKET interface, thereby achieving reliable data transmission.

[0237] (6) Communication service termination: When the communication service of all nodes ends and the application service ends, each service node in the big data node cluster releases all communication threads and process resources in the external communication service. Specifically, the external communication service process of all nodes in the big data node cluster releases the worker thread pool resources and all queue pairs to ensure the smooth recovery of thread and network card resources. Furthermore, the external communication service process is terminated to release process resources.

[0238] Figure 5 This is an example diagram of the data block transmission effect of the data transmission scenario provided in the embodiment of the present application, combined with Figure 5 As shown, the difference diagram of the transmission effect between the present data transmission scenario instance and the existing related technology is shown. Specifically, "1" represents the transmission effect (success rate of request) of the existing related technology, and "2" represents the transmission effect (success rate of request) of the present data transmission scenario instance. Figure 5 It can be obtained that by executing the above data transmission scenario instance, as the number of communication tasks of data blocks (i.e., the number of data block transmission requests) increases, the gap between the transmission effect of the related technology and the transmission effect of this data transmission scenario instance becomes larger and larger. For example, when the number of communication tasks is 1250 (1.25k), the transmission effect (success rate of requests) of the existing related technology exceeds 95%, and the transmission effect (success rate of requests) of this data transmission scenario instance is close to 100%. As the number of tasks increases, when the number of tasks reaches 20k, the transmission effect (success rate of requests) of the existing related technology is less than 85%, while the transmission effect (success rate of requests) of this data transmission scenario instance exceeds 90%. Under the big data processing framework, the success rate of data block acquisition requests can reach 90.6% to 99.6%. Therefore, this data transmission scenario instance meets the reliable transmission requirements of big data of application services.

[0239] By executing the above data transmission scenario examples, the following effects can be achieved: On the one hand, for users of application services, ensuring the high reliability of big data processing requests without any notice can improve user experience and the accuracy of user data analysis. On the other hand, for big data operation and maintenance technicians, ensuring high reliability of requests from the software level can reduce the operation and maintenance time cost caused by a large number of request failures, and automatically restore requests in the event of RDMA network anomalies. In addition, for big data applications themselves, the reliability transmission solution of this data transmission scenario example can significantly improve the success rate of applications.

[0240] As can be seen from the above, the embodiment of the present application forms a distributed system corresponding to the big data processing framework through multiple service nodes, and each service node is registered with the meta information of each data block stored on all service nodes. According to the meta information of each data block, the storage node address corresponding to each data block in the distributed system can be quickly located, so that there is no need to traverse and send data block acquisition requests to all service nodes in the distributed system, reducing network interactions and improving data acquisition efficiency. In addition, based on the remote direct memory access network, the data block can be remotely transmitted to other service nodes, thereby realizing the separation of data calculation and communication, avoiding the simultaneous operation of the calculation process and the communication process on a single service node, which makes the operation load of the service node too high, thereby avoiding the risk of data transmission failure and improving the reliability of data transmission. At the same time, for any two service nodes in the distributed system of the big data processing framework, a communication queue pair is established separately, and the communication queue pair can be reused for multiple data block transmissions between the two service nodes, avoiding frequent full connections between any two service nodes. In this way, while satisfying reliable transmission in big data processing, communication delays are further avoided, thereby improving data block transmission efficiency. In addition, when separating data calculation and communication, the computing resources of the currently used service nodes are utilized, and each service node can be used as both a computing end and a communication end, so there is no need to set up an additional communication server, reducing equipment costs.

[0241] The specific implementation of the above steps can be found in the previous embodiments, which will not be described in detail here.

[0242] In order to better implement the data transmission method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above data transmission method. The meanings of the terms are the same as those in the above data transmission method, and the specific implementation details can refer to the description in the method embodiment.

[0243] See also Figure 6 , Figure 6A structural schematic diagram of a data transmission device provided in an embodiment of the present application, wherein the data transmission device is integrated in a computer device of the present application, wherein the data transmission device may include an acquisition unit 401, a search unit 402, a loading unit 403 and a transmission unit 404.

[0244] An acquiring unit 401 is used to acquire a data block acquisition request sent by a target node, where the data block acquisition request carries a target data block identifier;

[0245] The target node and the local node belong to different service nodes in the distributed system. Different data blocks are stored on each service node in the distributed system, and meta information of each data block is registered on each service node. A corresponding communication queue pair is pre-created between every two different service nodes in the distributed system.

[0246] A search unit 402 is used to obtain target meta information associated with the target data block identifier, and search for the target data block associated with the target meta information from the local storage;

[0247] A loading unit 403, configured to determine a target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair;

[0248] The transmission unit 404 is used to transmit the target data block in the target communication queue pair to the target node through the remote direct memory access network.

[0249] In some implementations, the searching unit 402 is further configured to:

[0250] Determine the target storage node address corresponding to the target data block identifier according to the target meta information;

[0251] When it is identified that the target storage node address is consistent with the node address of the local node, the target data block corresponding to the target data block identifier is searched from the local storage.

[0252] In some implementations, the searching unit 402 is further configured to:

[0253] When it is identified that the target storage node address is inconsistent with the node address of the local node, the current time information is obtained, and data block abnormal information is generated according to the current time information and the node address of the local node;

[0254] Feedback data block exception information to the target node.

[0255] In some implementations, the loading unit 403 is further configured to:

[0256] The target data block in the local storage is transferred to the main storage space of the local node. The main storage space is used to store data that is directly read and written by the local processor;

[0257] The target data block in the main storage space is copied to the target communication queue pair.

[0258] In some implementations, the target communication queue pair includes a target sending queue and a target receiving queue, and the loading unit 403 is further configured to:

[0259] The target data block in the main storage space is copied to the target sending queue in the target communication queue pair.

[0260] In some implementations, the transmission unit 404 is further configured to:

[0261] Run the direct transfer function corresponding to the remote direct memory access network;

[0262] The target data block in the target communication queue pair is transmitted to the target node through the direct transmission function.

[0263] In some embodiments, the data transmission device further includes a detection unit, configured to:

[0264] Get the current transmission delay and packet loss rate of the remote direct memory access network;

[0265] The transmission unit 404 is further configured to:

[0266] When the transmission delay duration is less than a preset delay threshold and the packet loss rate is less than a preset packet loss rate threshold, the target data block in the target communication queue pair is transmitted to the target node through the remote direct memory access network.

[0267] In some implementations, the transmission unit 404 is further configured to:

[0268] When the transmission delay time is greater than or equal to the preset delay threshold, or the packet loss rate is greater than or equal to the preset packet loss rate threshold, the network address of the target node is obtained;

[0269] Establishing a data transmission channel between a local node and a target node of a network address through a socket interface;

[0270] When the target data block is queried from the main storage space of the local node, the target data block in the main storage space is transmitted to the target node corresponding to the network address through the data transmission channel;

[0271] When the target data block is not found in the main storage space, the target data block is loaded from the local storage into the main storage space, and the target data block in the main storage space is transmitted to the target node corresponding to the network address through the data transmission channel.

[0272] In some embodiments, the data transmission device further comprises a starting unit, configured to:

[0273] Run the remote task program on the local node;

[0274] Among them, the local node after running the remote task program and other service nodes running the remote task program together form a distributed system;

[0275] Start the local external communication service process on the local node after running the remote task program;

[0276] Through the local external communication service process, communication queue pairs are created with other service nodes in the distributed system.

[0277] In some embodiments, the startup unit is further used to:

[0278] Get the communication configuration parameters of the local node;

[0279] Allocate a target amount of memory resources to the local external communication service process according to the communication configuration parameters;

[0280] Get the service node address of each other service node in the distributed system;

[0281] Based on the target amount of memory resources, a communication queue pair between the local node and each service node address is created in the local external communication service process.

[0282] In some implementations, the metadata of the data block is registered in an external communication service process running on each service node, and the search unit 402 is further used to:

[0283] According to the target data block identifier, the corresponding target meta information is searched from the local external communication service process of the local node.

[0284] In some implementations, the metadata of the data block is registered in an external communication service process running on each service node, and the search unit 402 is further used to:

[0285] According to the target data block identifier, the corresponding target meta information is searched from the local external communication service process of the local node.

[0286] In some embodiments, the data transmission device further includes a registration unit, configured to:

[0287] Obtaining meta information of a data block pre-stored by each service node in the distributed system for a target business service, the meta information including a data block identifier and a storage node address;

[0288] Each piece of meta information is registered in the local external communication service process, so that the local external communication service process records the storage node address associated with each data block identifier.

[0289] In some embodiments, the data transmission device further includes a data block acquisition unit, configured to:

[0290] Determine the business data block identifier obtained by the local node for the target business service request;

[0291] Based on the local external communication service process, query the remote node address corresponding to the business data block identifier;

[0292] Sending a business data block acquisition request to a target service node corresponding to the remote node address in the distributed system, where the data block acquisition request carries a business data block identifier;

[0293] The business data block corresponding to the business data block identifier fed back by the target service node is obtained through the target receiving queue in the communication queue pair pre-created between the local node and the target service node, so as to run the target business service according to the business data block.

[0294] As can be seen from the above, the embodiment of the present application forms a distributed system corresponding to the big data processing framework through multiple service nodes, and each service node is registered with the meta information of each data block stored on all service nodes. According to the meta information of each data block, the storage node address corresponding to each data block in the distributed system can be quickly located, so that there is no need to traverse and send data block acquisition requests to all service nodes in the distributed system, reducing network interactions and improving data acquisition efficiency. In addition, based on the remote direct memory access network, the data block can be remotely transmitted to other service nodes, thereby realizing the separation of data calculation and communication, avoiding the simultaneous operation of the calculation process and the communication process on a single service node, which makes the operation load of the service node too high, thereby avoiding the risk of data transmission failure and improving the reliability of data transmission. At the same time, for any two service nodes in the distributed system of the big data processing framework, a communication queue pair is established separately, and the communication queue pair can be reused for multiple data block transmissions between the two service nodes, avoiding frequent full connections between any two service nodes. In this way, while satisfying reliable transmission in big data processing, communication delays are further avoided, thereby improving data block transmission efficiency. In addition, when separating data calculation and communication, the computing resources of the currently used service nodes are utilized, and each service node can be used as both a computing end and a communication end, so there is no need to set up an additional communication server, reducing equipment costs.

[0295] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0296] See also Figure 7 , Figure 7The schematic diagram of the structure of the terminal provided in the embodiment of the present application includes a structural frame of a part of the terminal 110 implementing the embodiment of the present application. The terminal may be Figure 1 The terminal 110 shown in the figure specifically includes: a radio frequency (RF) circuit 510, a memory 515, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590. Those skilled in the art will appreciate that the structure of the terminal 110 shown in the figure does not limit the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0297] The RF circuit 510 may be used for receiving and sending signals during information transmission or communication. In particular, after receiving downlink information from the base station, the information is sent to the processor 580 for processing. In addition, the uplink data is sent to the base station.

[0298] The memory 515 may be used to store software programs and modules. The processor 580 executes various functional applications and data transmission of the terminal by running the software programs and modules stored in the memory 515 .

[0299] The input unit 530 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532 .

[0300] The display unit 540 may be used to display input information or provided information and various menus of the terminal. The display unit 540 may include a display panel 541 .

[0301] The audio circuit 560 , the speaker 561 , and the microphone 562 may provide an audio interface.

[0302] In this embodiment, the processor 580 included in the terminal 110 can execute the data transmission method of the previous embodiment.

[0303] See also Figure 8 , Figure 8 The schematic diagram of the structure of the server provided in the embodiment of the present application includes a structural frame of the server 120 implementing the embodiment of the present application. The server may be Figure 1The server 120 shown in the figure may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 622 (e.g., one or more processors) and memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 may be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server 120. Furthermore, the central processing unit 622 may be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the server 120.

[0304] The server 120 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0305] The central processor 622 in the server 120 can be used to execute the data transmission method of the embodiment of the present application.

[0306] An embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the data transmission methods of the aforementioned embodiments.

[0307] The embodiment of the present application further provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes and implements the above-mentioned data transmission method.

[0308] In addition, the terms "comprises" and "includes" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product or apparatus.

[0309] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0310] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0311] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0312] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0313] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0314] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0315] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0316] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0317] The above is a specific description of the implementation method of the present application, but the present application is not limited to the above-mentioned implementation method. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data transmission method, characterized in that: include: Obtaining a data block acquisition request sent by a target node, wherein the data block acquisition request carries a target data block identifier; The target node and the local node belong to different service nodes in the distributed system, an external communication service process is independently run on each service node, different data blocks are stored on each service node in the distributed system, and meta information of each data block is registered on each service node, and a corresponding communication queue pair is pre-created between every two different service nodes in the distributed system; Acquire target meta information associated with the target data block identifier, and search for the target data block associated with the target meta information from local storage; Determine a target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair; transmitting the target data block in the target communication queue pair to the target node via a remote direct memory access network; The step of searching the local storage for the target data block associated with the target meta-information includes: Determine, according to the target meta information, a target storage node address corresponding to the target data block identifier; When it is identified that the target storage node address is consistent with the node address of the local node, searching the local storage for the target data block corresponding to the target data block identifier; When it is identified that the target storage node address is inconsistent with the node address of the local node, the current data block acquisition request is forwarded to each service node in the distributed system to receive the target data block storage address fed back by other target service nodes, and the target metadata in the local external communication service process running on the local node is updated according to the target data block storage address to obtain the updated target metadata, and the target data block sent by other target service nodes is received.

2. The method according to claim 1, characterized in that The method further comprises: When it is identified that the target storage node address is inconsistent with the node address of the local node, current time information is acquired, and data block exception information is generated according to the current time information and the node address of the local node; Feedback the data block exception information to the target node.

3. The method according to any one of claims 1 to 2, characterized in that: The step of loading the target data block into the target communication queue pair comprises: The target data block in the local storage is transferred to the main storage space of the local node, where the main storage space is used to store data directly read and written by the local processor; The target data block in the main storage space is copied to the target communication queue pair.

4. The method according to claim 3, characterized in that: The target communication queue pair includes a target sending queue and a target receiving queue, and the copying of the target data block in the main storage space to the target communication queue pair includes: The target data block in the main storage space is copied to the target sending queue in the target communication queue pair.

5. The method according to claim 1, characterized in that The step of transmitting the target data block in the target communication queue pair to the target node through a remote direct memory access network includes: Run the direct transfer function corresponding to the remote direct memory access network; The target data block in the target communication queue pair is transmitted to the target node through the direct transmission function.

6. The method according to claim 5, characterized in that Before transmitting the target data block in the target communication queue pair to the target node through the remote direct memory access network, the method further comprises: Get the current transmission delay and packet loss rate of the remote direct memory access network; Then, transmitting the target data block in the target communication queue pair to the target node through the remote direct memory access network includes: When the transmission delay duration is less than a preset delay threshold and the packet loss rate is less than a preset packet loss rate threshold, the target data block in the target communication queue pair is transmitted to the target node via the remote direct memory access network.

7. The method according to claim 6, characterized in that The method further comprises: When the transmission delay time is greater than or equal to a preset delay threshold, or the packet loss rate is greater than or equal to a preset packet loss rate threshold, acquiring the network address of the target node; Establishing a data transmission channel between the local node and the target node of the network address through a socket interface; When the target data block is found in the main storage space of the local node, the target data block in the main storage space is transmitted to the target node corresponding to the network address through the data transmission channel; When the target data block is not found in the main storage space, the target data block is loaded from the local storage into the main storage space, and the target data block in the main storage space is transmitted to the target node corresponding to the network address through the data transmission channel.

8. The method according to claim 1, characterized in that Before obtaining the data block acquisition request sent by the target node, the method further includes: Running the remote task program on the local node; The local node running the remote task program and other service nodes running the remote task program together form a distributed system; Starting a local external communication service process on the local node after running the remote task program; Through the local external communication service process, communication queue pairs are respectively created with other service nodes in the distributed system.

9. The method according to claim 8, characterized in that The step of creating communication queue pairs with other service nodes in the distributed system respectively through the local external communication service process includes: Acquiring communication configuration parameters of the local node; Allocating a target amount of memory resources to the local external communication service process according to the communication configuration parameters; Obtaining a service node address of each other service node in the distributed system; Based on the target amount of memory resources, a communication queue pair between the local node and each service node address is respectively created in the local external communication service process.

10. The method according to claim 1 or 8, characterized in that: The metadata of the data block is registered in an external communication service process running on each service node, and the step of obtaining target metadata associated with the target data block identifier includes: According to the target data block identifier, corresponding target meta information is searched from the local external communication service process of the local node.

11. The method according to claim 10, characterized in that Before searching the local external communication service process of the local node for the corresponding target meta information according to the target data block identifier, the method further includes: Obtaining meta information of a data block pre-stored by each service node in the distributed system for a target business service, the meta information including a data block identifier and a storage node address; Each piece of meta information is registered in the local external communication service process, so that the local external communication service process records the storage node address associated with each data block identifier.

12. The method according to claim 1, characterized in that The method further comprises: Determine the business data block identifier obtained by the local node for the target business service request; Based on the local external communication service process, query the remote node address corresponding to the business data block identifier; Sending a service data block acquisition request to a target service node corresponding to the remote node address in the distributed system, wherein the data block acquisition request carries a service data block identifier; The business data block corresponding to the business data block identifier fed back by the target service node is obtained through the target receiving queue in the communication queue pair pre-created between the local node and the target service node, so as to run the target business service according to the business data block.

13. A data transmission device, characterized in that: include: An acquisition unit, configured to acquire a data block acquisition request sent by a target node, wherein the data block acquisition request carries a target data block identifier; The target node and the local node belong to different service nodes in the distributed system, an external communication service process is independently run on each service node, different data blocks are stored on each service node in the distributed system, and meta information of each data block is registered on each service node, and a corresponding communication queue pair is pre-created between every two different service nodes in the distributed system; A search unit, configured to obtain target meta information associated with the target data block identifier, and search for the target data block associated with the target meta information from a local storage; a loading unit, configured to determine a target communication queue pair pre-created by the local node for the target node, and load the target data block into the target communication queue pair; A transmission unit, configured to transmit the target data block in the target communication queue pair to the target node via a remote direct memory access network; Among them, the search unit is also used to determine the target storage node address corresponding to the target data block identifier based on the target metadata; when it is identified that the target storage node address is consistent with the node address of the local node, the target data block corresponding to the target data block identifier is searched from the local storage; when it is identified that the target storage node address is inconsistent with the node address of the local node, the current data block acquisition request is forwarded to each service node in the distributed system to receive the target data block storage address fed back by other target service nodes, and the target metadata in the local external communication service process running on the local node is updated according to the target data block storage address to obtain the updated target metadata, and the target data block sent by other target service nodes is received.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data transmission method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the data transmission method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data transmission method according to any one of claims 1 to 12 is implemented.

Images