A method for data transmission between multi-processing cards

Abstract

The embodiment of the specification provides a data transmission method between multiple processing cards. The multiple processing cards have direct channels and indirect channels therebetween, and the indirect channels include two or more direct channels. The multiple processing cards are located on a mainboard of a same processing device. The method comprises the following steps: receiving a card-to-card data transmission request; determining two or more channels between a first processing card corresponding to a source address and a second processing card corresponding to a target address; selecting one or more target channels from the two or more channels for transmitting to-be-transmitted data; obtaining one or more data subsets of the to-be-transmitted data based on the number of the target channels and the to-be-transmitted data; transmitting the one or more data subsets through the one or more target channels; and based on the one or more data subsets, initiating a data transmission instruction to a processing card related to the target channel, transmitting corresponding data subsets through the processing card related to the target channel, and then transmitting the to-be-transmitted data to the target address.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese application filed on June 15, 2023, with application number 202310707879.2 and invention title "A method and system for data transmission between multiple processing cards". Technical Field

[0003] This specification relates to the field of computer technology, and in particular to a method for data transmission between multiple processors or multiple processing cards. Background Technology

[0004] A multiprocessor device is a processing device containing multiple processors. Each processor has the ability to perform data processing, can exchange data with each other, is managed by a unified operating system, and can share I / O devices, disks, and other peripherals. Generally, in a multiprocessor device, multiple processors reside on the same motherboard. The motherboard has wiring and slots, and the processors are attached to these slots as cards; therefore, a processor can also be called a processing card.

[0005] Multiprocessor devices offer significantly more powerful data processing capabilities than single-processor devices and are typically used in industrial computing environments to handle large-scale data computations. In big data computing scenarios, frequent and extensive data exchange between multiple processors is crucial; therefore, optimizing data transfer between processors is a pressing issue that needs to be addressed. Summary of the Invention

[0006] This specification provides one or more embodiments of a data transmission method between multiple processing cards. The method includes: receiving a call from an application process via a data transmission client and initiating an inter-card data transmission request to a data transmission service process. The inter-card data transmission request includes a target address and a source address, with the target address and source address corresponding to different processing cards; selecting one or more target channels from two or more channels between a first processing card corresponding to the source address and a second processing card corresponding to the target address via the data transmission service process, and returning the target channel identifier to the data transmission client; obtaining one or more data subsets of the data to be transmitted based on the number of target channels and the data to be transmitted via the data transmission client; initiating a data transmission instruction to a transmission process on the processing card associated with the target channel based on the one or more data subsets via the data transmission client; and transmitting the corresponding data subsets via the transmission process on the processing card associated with the target channel, thereby transmitting the data to be transmitted to the target address.

[0007] This specification provides one or more embodiments of a data transmission system between multiple processing cards. The system includes: a data transmission client, invoked by an application process to initiate an inter-card data transmission request to a data transmission server. The inter-card data transmission request includes a target address and a source address, which correspond to different processing cards; a data transmission server, configured to respond to the inter-card data transmission request, select one or more target channels from two or more channels between a first processing card corresponding to the source address and a second processing card corresponding to the target address, and return the target channel identifier to the data transmission client; the data transmission client is further configured to obtain one or more data subsets of data to be transmitted based on the number of target channels and the data to be transmitted, and, based on one or more data subsets, initiate a data transmission instruction to a transmission module on the processing card associated with the target channel; and a transmission module, configured to respond to the data transmission instruction, transmit the corresponding data subset, and thus transmit the data to be transmitted to the target address.

[0008] This specification provides one or more embodiments of a storage medium for storing computer instructions, which, when at least a portion of the computer instructions are executed by a processor or processing card, implement the aforementioned data transmission method between multiple processing cards.

[0009] This specification provides an apparatus in one or more embodiments, including multiple processing cards and a storage medium. The multiple processing cards have direct connection channels and / or indirect channels, and the indirect channels include two or more direct connection channels. The storage medium stores computer instructions, and the multiple processing cards are used to execute the computer instructions to implement the aforementioned data transmission method between multiple processing cards.

[0010] This specification provides one or more embodiments of a data transmission method between multiple processing cards, wherein the multiple processing cards have direct connection channels and / or indirect channels; the method includes: receiving an inter-card data transmission request; the inter-card data transmission request includes a target address and a source address, the target address and the source address corresponding to different processing cards; determining two or more channels between a first processing card corresponding to the source address and a second processing card corresponding to the target address; obtaining a weight value for each of the two or more channels; the weight value is positively correlated with the data transmission bandwidth of the channel and negatively correlated with the observed load of the channel; based on the weight value, selecting one or more target channels from the two or more channels for transmitting data to be transmitted. Attached Figure Description

[0011] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0012] Figure 1This is a schematic diagram of a multiprocessor motherboard according to some embodiments of this specification;

[0013] Figure 2 This is an exemplary flowchart of a data transfer method between multiple processing cards according to some embodiments of this specification;

[0014] Figure 3 This is an exemplary flowchart illustrating the determination of a target channel according to some embodiments of this specification;

[0015] Figure 4 These are example diagrams of component tables shown according to some embodiments of this specification;

[0016] Figure 5 This is an example diagram of a load table shown according to some embodiments of this specification;

[0017] Figure 6 This is an exemplary schematic diagram of a data subset caching method on a transit processing card according to some embodiments of this specification;

[0018] Figure 7 This is an exemplary schematic diagram of a data subset caching method on a transit processing card according to other embodiments of this specification;

[0019] Figure 8 This is a schematic block diagram of a data transmission system between multiple processing cards according to some embodiments of this specification. Detailed Implementation

[0020] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0021] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0022] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0023] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0024] In big data processing scenarios, multiprocessor devices are commonly used. A multiprocessor device refers to a device containing multiple processors. Each processor has data processing capabilities, can exchange data with others, is managed by a unified operating system, and can share I / O devices, disks, and other peripherals. Generally, multiple processors reside on the same motherboard, which has wiring and slots. The processors are attached to these slots as cards. In some scenarios, the terms "processor," "processor card," and "processor card" are used interchangeably.

[0025] Processors located on the same processing device or motherboard can be divided into main processors and coprocessors. The main processor is the processing core of the device, typically implemented by a general-purpose CPU (Center Processing Unit). Coprocessors receive scheduling from the main processor and assist it in performing specific computational tasks. Depending on the specific computational task they are responsible for, coprocessors can be categorized as math coprocessors, graphics coprocessors, etc. Math coprocessors are used for numerical processing or computation, also known as floating-point coprocessors, and their numerical operations are faster than those of the main processor. Graphics processors, also known as GPUs (Graphics Processing Units), are used for graphics display and computational acceleration, featuring high-speed parallel computing and suitable for computational tasks that can be processed in parallel.

[0026] In some embodiments, multiprocessor devices can be used for machine learning model training or prediction. Specifically, when there are many training samples, distributed training can be implemented on multiple processors. For example, the main processor can distribute the same initial model to multiple coprocessors, each coprocessor using a portion of the training samples to train the initial model. As an example, each coprocessor can use the initial model to process the feature information of the training samples, obtain prediction results, and return the prediction results to the main processor. The main processor obtains the loss function value based on each prediction result and the label of the training sample, and then obtains the average gradient value, returning the average gradient value to each coprocessor. The coprocessors update the parameters of their local models based on the average gradient value, and so on, performing multiple rounds of iterative distributed training to obtain the trained model. Compared to single-processor processing devices, the model training time is greatly reduced because multiple coprocessors train the model in parallel. When the model size is large, distributed prediction can be implemented on multiple processors. For example, the main processor can split the trained model into multiple sub-models and distribute them to multiple coprocessors. Simultaneously, it can vertically segment the feature terms of the object to be predicted and distribute them to the corresponding coprocessors. Each coprocessor uses its local sub-model to process the feature values ​​of the corresponding feature terms of the object to be predicted, obtaining a local prediction result. The coprocessor then returns the local prediction result to the main processor, which aggregates all the local prediction results to obtain the final prediction result. It can be seen that during large-scale model training or prediction, there is a large amount of frequent data interaction between processors.

[0027] It should be noted that distributed training / prediction of models is an application scenario of multiprocessor devices and should not be understood as a limitation of the application scenarios of multiprocessor devices. In some embodiments, multiprocessor computing scenarios are also applicable to knowledge graph queries, etc.

[0028] Figure 1 This is a schematic diagram of a multiprocessor motherboard according to some embodiments of this specification. For example... Figure 1As shown, the motherboard 100 includes two CPU boards and eight GPU boards, with data cables or data channels connecting the CPU boards. CPU0 and CPU1 can be considered the main processors, and GPUs 0-7 can be considered coprocessors. The CPU and GPUs can be directly connected via a PCIe interface, with a transmission bandwidth of 16GB / s (G Byte / second) or 32GB / s. GPUs can be directly connected via an NVLink interface, which offers a higher transmission bandwidth than PCIe, for example, reaching 150GB / s or more. CPUs can be directly connected via interfaces such as QPI or UPI, with a transmission bandwidth between PCIe and NVLink. In some embodiments, two or more GPUs can be grouped together (e.g., GPU0 and GPU1 in the same dashed box) and connected to the CPU via a single PCIe direct connection channel. The CPU can then use "addressing" to transmit different information to GPU0 or GPU1 respectively. There can be two or more direct connection channels between two GPUs; in this case, the bandwidth between the two GPUs is the sum of the bandwidths of the two or more direct connection channels.

[0029] according to Figure 1 It can be seen that two processor boards can exchange data not only through direct connections but also through indirect connections consisting of two (or more) direct connections. For example, CPU0 to GPU0 has one PCIe direct connection, and can also exchange data via indirect connections such as CPU0-GPU3-GPU0 or CPU0-CPU1-GPU6-GPU0. When two processors need to exchange data, how to select the appropriate target channel from multiple channels to improve data transfer speed or balance data transfer load becomes a problem worthy of research.

[0030] To this end, some embodiments of this specification propose a global multi-channel data transmission acceleration method across multiprocessor cards and / or across tasks, including multiple intrinsic optimization steps such as multi-channel optimization selection, data segmentation, concurrent transmission, and application layer transparency, which ultimately significantly accelerates the data transmission performance between processing cards.

[0031] Figure 2This is an exemplary flowchart illustrating a data transmission method between multiple processing cards according to some embodiments of this specification, which implements data transmission in a CS (Client-Server) architecture. In some embodiments, the multiple processing cards may include a main processor CPU board and multiple coprocessor GPU boards. The server is implemented by an independent process running on the CPU board. Each processor board has a transmission process deployed on it for implementing data transmission. The client is invoked by an application (or application process) to initiate requests to the server process or send data transmission instructions to the transmission process on the corresponding processing card.

[0032] An application process, or application program, is responsible for data computation or information processing at the application layer, such as using a model to process feature data to obtain prediction results or training a model based on training samples. When an application process obtains a computation result (which can be an intermediate or final result), it needs to transmit the result to other application processes located on different processing cards for subsequent computation, thus triggering data transmission between processing cards. In some embodiments of this specification, when an application process needs to transmit data to other processing cards, it can call a data transmission client to initiate an inter-card data transmission request to the data transmission service process. Correspondingly, the data transmission client can be deployed on each processing card for application processes to call. Figure 2 As shown, the data transmission method 200 provided in some embodiments of this specification may specifically include the following steps:

[0033] Step 210: Receive the application process call through the data transmission client, and then initiate an inter-card data transmission request to the data transmission service process.

[0034] The data transmission client can be encapsulated as a function, whose calling interface includes a function name and input parameters. The input parameters of the data transmission client can include the source address and destination address of the data to be transmitted. The source address can specifically be the starting address of the storage area where the data to be transmitted is located. When the application is located on a CPU board, the source address can point to a storage area in the processing device's memory or a storage area on the disk. When the application is located on a GPU board, the source address can point to a storage area in the GPU's video memory. The destination address is the storage area where the data to be transmitted will reach, pointing to a storage area on a processing card different from the aforementioned application. Similar to the source address, the specific storage location can be memory, disk, or video memory, etc. In some embodiments, the processing card corresponding to the source address can be called the source processing card or the first processing card, and the processing card corresponding to the destination address can be called the target processing card or the second processing card. In some embodiments, the input parameters can also include a data offset, which reflects the amount of data to be transmitted, such as 1024B (Byte), 320B, etc. Based on the starting address and offset of the data to be transmitted, the data transmission client can determine the address of each byte of data in the corresponding storage area of ​​the source processing card.

[0035] The application can invoke the data transmission client through its API. The data transmission client obtains the source and destination addresses of the data to be transmitted. The client can then initiate an inter-card data transmission request to the service process, which may include the source and destination addresses of the data to be transmitted. Since the data transmission client resides within the application process, it can transmit the inter-card data transmission request to the data transmission service process via inter-process communication methods such as pipes and sockets.

[0036] Step 220: Select one or more target channels from two or more channels between the first processing card corresponding to the source address and the second processing card corresponding to the target address through the data transmission service process, and return the target channel identifier to the data transmission client.

[0037] In some embodiments, the data transmission service process can follow Figure 3 The process shown in step 300 determines the target channel. For example... Figure 3 As shown, process 300 includes:

[0038] Step 310: Determine two or more channels between the first processing card corresponding to the source address and the second processing card corresponding to the target address.

[0039] As mentioned above, there are multiple channels between any two processing cards in the multiprocessor system, including one or more direct channels and one or more indirect channels. In some embodiments, the data transmission service process can determine the corresponding processing card based on the source address and destination address in the inter-card data transmission request. Specifically, the source address may include the processing card number and the starting address of the storage area (such as memory, register, or video memory) of the data to be transmitted on the processing card, and the destination address is similar. Alternatively, each storage address is globally unique among the multiprocessor systems; therefore, the source address or destination address can directly determine which processing card it points to. Subsequently, the data transmission service process can determine all channels between the first processing card corresponding to the source address and the second processing card corresponding to the destination address.

[0040] In some embodiments, the data transfer service process may obtain a component table that records the direct connection channels between multiple processing cards and the data transfer bandwidth. Figure 4 This specification provides a list of components for some embodiments shown in the present specification, such as... Figure 4 As shown, Table 400 records the processing cards with direct connection channels among the multiple processing cards, and the data transmission bandwidth of each direct connection channel. Table 400 shows that there is a direct connection channel between the CPU and GPU0 with a data transmission bandwidth of 12GB / s, and a direct connection channel between GPU0 and GPU1 with a data transmission bandwidth of 24GB / s. Table 400 also records the data transmission bandwidth within each processing card; for example, the data transmission bandwidth within GPU0 is as high as 1440GB / s. Since an indirect channel consists of two or more direct connection channels, based on the component table, the data transmission service process can also determine the indirect channel between any two processing cards, and thus determine all channels between the first processing card and the second processing card. The direct connection channels between processing cards are fixed; therefore, the data transmission service process only needs to obtain the component table once.

[0041] In some embodiments, the hardware driver of the motherboard or processing card can provide a component table lookup API (Application Programming Interface), which the data transfer service process can call to obtain the component table.

[0042] Step 320: Obtain the weight value of each of the two or more channels.

[0043] To improve transmission rates, the channel with the larger data transmission bandwidth can be selected from two or more channels between the first and second processing cards for data transmission. In actual use, the data transmission efficiency of a channel is not only related to bandwidth but also constrained by the observed load on the channel. The observed load can be the amount of data being transmitted and / or waiting to be transmitted on the channel, reflecting the actual occupancy of the channel. Therefore, the data transmission service process can obtain a weight value for each channel based on the data transmission bandwidth and the observed load for selection.

[0044] In some embodiments, the data transmission service process can determine the data transmission bandwidth of the direct connection channel between the first processing card and the second processing card based on Table 400. For the indirect channel between the two cards, the data transmission bandwidth of each direct connection channel on the indirect channel can be obtained, and the minimum data transmission bandwidth among them can be taken as the data transmission bandwidth of that indirect channel. Thus, the data transmission service process can obtain the data transmission bandwidth of each channel between the first processing card and the second processing card.

[0045] In some embodiments, the data transfer service process can obtain a load table that records the observed load on the direct connection channels between multiple processing cards. The observed load changes over time; therefore, the data transfer service process needs to obtain the load table each time it is used. Similar to the component table, the data transfer service process can obtain the load table by calling an API provided by the motherboard or processing card. Figure 5 As shown in the load table 500 of some embodiments of this specification, it can be determined from the table 500 that the current observation load on the direct connection channel between the CPU and GPU0 is 1GB, and the observation load on the direct connection channel between GPU0 and GPU1 is 4GB.

[0046] Based on the load table, the data transmission service process can determine the observed load of all direct-connect channels between the first and second processing cards. For indirect channels between the two cards, the observed load of each direct-connect channel on the indirect channel can be obtained. By summing the observed loads of all direct-connect channels, the observed load of that indirect channel can be obtained. Therefore, the data transmission service process can obtain the observed load of each channel between the first and second processing cards.

[0047] In some embodiments, the data transmission service process can determine the weight value of each channel based on its data transmission bandwidth and observation load. Specifically, the weight value can be positively correlated with the channel's data transmission bandwidth and negatively correlated with the observation load. In some embodiments, the data transmission service process can determine a first weight value based on the data transmission bandwidth, which is positively correlated with the data transmission bandwidth, and determine a second weight value based on the observation load, which is negatively correlated with the observation load. Finally, the data transmission service process can combine the first weight value and the second weight value to obtain the weight value of the channel, for example, by adding or multiplying the two values.

[0048] Step 330: Based on the weight value, select one or more target channels from the two or more channels to transmit the data to be transmitted.

[0049] In some embodiments, the data transmission service process can sort the channels in the first processing card and the second processing card in descending order based on weight values, and select the top n channels as target channels. n can be set according to actual needs, such as 1, 2 or 3.

[0050] The data transmission service process can return the identifier of the target channel to the data transmission client. In some embodiments, the channel between processing cards can have a unique identifier. Specifically, the identifier can include a string or a number. In some embodiments, the channel identifier can consist of the numbers of the processing cards involved, such as an indirect channel CPU0-GPU3-GPU0, a direct channel CPU0-GPU0. When there are two or more direct channels between two cards, the channel identifier can also include the direct channel identifier, such as direct channels GPU1-GPU2-1, GPU1-GPU2-2, where the single value "1" or "2" is the direct channel identifier. Correspondingly, the indirect channel identifier can be CPU0-(GPU2-GPU1-1) or CPU0-(GPU2-GPU1-2). Parentheses can also be omitted.

[0051] Step 230: The data transmission client obtains one or more data subsets of the data to be transmitted based on the number of target channels and the data to be transmitted.

[0052] The data transmission client can split the data to be transmitted into more than one data subset based on the data of the target channel. In some embodiments, the data subsets can correspond one-to-one with the target channels. When the number of target channels is 1, the number of data subsets is 1, that is, the data to be transmitted is not split. In some embodiments, the inter-card data transmission request can also include the offset of the storage area of ​​the data to be transmitted relative to the source address. The offset is the amount of data to be transmitted. In this case, the data transmission client can determine the amount of data in each data subset based on the offset. As an example, the amount of data in the data subset corresponding to the target channel with a larger data transmission bandwidth can be set to be larger. For example, if the offset is 1280B, the number of target channels is 3, and their bandwidths are 48GB / s, 12GB / s, and 12GB / s respectively, the data to be transmitted can be split into 3 data subsets with amounts of 1000B, 140B, and 140B respectively.

[0053] The data transmission client can obtain the starting address and offset of each data subset based on the source address and the amount of data in the subset. Continuing the previous example, if the source address is 0xB13FF (hexadecimal), then the starting address of the first data subset is 0xB13FF, and the offset is 1000B. The starting address of the second data subset is 0xB13FF+0x3E8 (hexadecimal representation of 1000), and the offset is 140B. The starting address of the third data subset is 0xB13FF+0x3E8 (hexadecimal representation of 1000)+0x8C (hexadecimal representation of 140), and the offset is 140B.

[0054] Step 240: The data transmission client initiates a data transmission command to the transmission process on the processing card associated with the target channel based on the one or more data subsets.

[0055] In some embodiments, the data transmission client can communicate with the transmission process on the processing card associated with the target channel and send data transmission instructions to it. The data transmission instructions may include the starting address of a data subset, the subset offset, and the target address. In some embodiments, the data transmission instructions may also include the amount of data to be transmitted (and / or the number of data subsets) and the identifiers of the data subsets. The identifiers of the data subsets may indicate the storage order of each data subset in the storage area corresponding to the source address; for example, data subsets stored on the first processing card that are closer to the source address may have smaller sequence numbers.

[0056] For a directly connected target channel, the data transmission client can directly communicate with the transmission process on the second processing card and transmit the data transmission command to the transmission process of the second processing card.

[0057] For indirect target channels, the data transmission client can communicate with the nearest processing card on the channel and send data transmission instructions to that processing card. The nearest processing card can be the processing card in the target channel directly connected to the first processing card. Taking the target channel CPU0-GPU3-GPU1 as an example, the first processing card is CPU0, and its nearest processing card is GPU3. In some embodiments, the data transmission client can also transmit relay routing information to the nearest processing card. The relay routing information reflects the processing cards the data subset passes through before reaching the target address. In some embodiments, the data transmission client can directly transmit the identifier of the target channel, composed of the numbers of the processing cards passed through, to the transmission process of the nearest processing card. The transmission process of the nearest processing card can read the data subset from the first processing card to its local machine based on the data transmission instructions, and then initiate data transmission instructions to the transmission processes of the next processing card on the target channel based on the relay routing information, and so on. The data subset jumps sequentially on the relay processing cards on the target channel until it finally reaches the target address. More details about the transmission process transmitting the data subset can be found in the relevant description of step 250.

[0058] Step 250: The corresponding data subset is transmitted through the transmission process on the processing card associated with the target channel, and then the data to be transmitted is transmitted to the target address.

[0059] After receiving a data transmission instruction, the transmission process on the processing card can locate the storage area of ​​the data subset corresponding to the previous processing card based on the starting address and offset of the data subset in the instruction, and read the data from it into the storage area of ​​this processing card.

[0060] When the target channel is a direct connection channel, the transmission process on the second processing card can directly read a subset of data from the storage area corresponding to the first processing card and store it in the storage area pointed to by the target address.

[0061] When the target channel is an indirect channel, the processing cards on the channel other than the first and second processing cards can be called relay processing cards. For relay processing cards, in order to transmit a data subset, they need to read the data subset from the storage area corresponding to the previous processing card and cache it locally before initiating a data transmission command to the next processing card. In some embodiments, the transmission process on the relay processing card can store the data subset in the global storage area or the temporary storage area of ​​the transmission process. The temporary storage area occupies the computing resources associated with the transmission process, which can be the registers corresponding to the transmission process. When the transmission process exits, the temporary storage area will also be released. The global storage area is relative to the temporary storage area and occupies the memory or video memory area. The transmission performance is basically the same whether the transmission process chooses to store data in the temporary storage area or cache data in the global storage area. The main difference lies in the type of resources occupied. In some embodiments, the transmission process can pre-determine the occupancy of the temporary storage area and dynamically select it. For example, when the free amount in the temporary storage area is greater than a set threshold, the data can be cached in the temporary storage area; otherwise, the data is cached in the global storage area.

[0062] In some embodiments, when the transmission process determines to cache a subset of data in the global storage area, it may open two or more buffer areas of preset size in the global storage area. The size of the buffer area may be fixed, such as 64B, 128B, etc. For each buffer area, the transmission process may open a corresponding task, which is used to read a portion of the data in the data subset and cache it in its corresponding buffer area. In some embodiments, the tasks may be executed in parallel or at intervals of a certain time difference.

[0063] Executing multiple tasks in parallel can be considered as multiple tasks executing independently, unaffected by the execution status of other tasks. (Combined) Figure 6Assume there are two buffer regions, Buffer1 and Buffer2, each 64 bytes in size. The data subset size is 320 bytes. The target channel is CPU0-GPU3-GPU0, and the current intermediary processing card is GPU3. The transmission process creates two tasks, s1 and s2, for each buffer region. Task s1 reads no more than 64 bytes of data (e.g., 64 bytes) from CPU0 into Buffer1 on GPU3, based on the starting address and offset of the data subset. Task s2 reads no more than 64 bytes of data from the remaining portion of the data subset into Buffer2. For example, s2 can read 64 bytes of data from the position (starting address + 64 bytes) into Buffer2. In some embodiments, after buffering data into the buffer region, the task can initiate a data transfer instruction to the transmission process of the next processing card (e.g., the second processing card GPU0). The data transfer instruction can include the address of the buffer region and the amount of data contained therein. After the transmission process of the next processing card reads the data from the buffer region, the task can continue to read data from the remaining portion of the data subset into the buffer region. For example, s1 can read 64B of data from the address (starting address + 64B + 64B) into Buffer1, and s2 can read 64B of data from the address (starting address + 64B + 64B + 64B) into Buffer2. Tasks s1 and s2 are executed in parallel until all 320B of data in the data subset has passed through the buffer area of ​​the current intermediate processing card to the next processing card. The process of data transfer from the CPU to the GPU can be viewed as H2D, and the process of data transfer between different GPUs can be viewed as D2D.

[0064] Executing multiple tasks at intervals of a certain time can be considered as the initiation of one task depending on the execution status of another task. Specifically, one task may be triggered only after a certain period of time has elapsed since the first task began execution. In some embodiments, a task may generate an event flag, such as event1, after a certain period of execution, or set the event flag to a valid state, such as "ready." The other task continuously queries the event flag, and only executes when its status is valid. (Combined with...) Figure 7 Continuing from the previous example, each task handles the same transactions as before, the only difference being... Figure 7 After storing data in Buffer1, task s1 generates an event flag. Task s2 continuously queries this event flag, and only executes when its status is valid. Multiple tasks are executed sequentially in this "pipeline" manner, which can reduce the bandwidth consumption between processing cards. That is, the amount of data transmitted on the same channel (such as the direct connection channel from CPU0 to GPU3) at the same time is less than that in the parallel method.

[0065] When there are two or more relay processing cards in the target channel, each relay processing card can read a subset of data from the previous processing card into its local buffer area in the manner described above, and then proceed to the next processing card.

[0066] For the second processing card in the indirect target channel, i.e., the processing card corresponding to the target address, its transmission process can read data from the buffer area in the previous relay processing card to the target address after receiving the data transmission instruction. In some embodiments, the transmission process on the second processing card can also open multiple tasks, which are executed in parallel or pipelined under the trigger of the transmission task of the previous processing card (similar to the transmission process on the relay processing card), reading the data in the corresponding buffer area into the storage area corresponding to the target address.

[0067] In some embodiments, the data to be transmitted is divided into two or more data subsets and transmitted to the second processing card through two or more target channels. At this time, the transmission process of the second processing card can store the data subsets from different target channels sequentially into the storage area corresponding to the target address, until all the data to be transmitted arrives at the storage area corresponding to the target address. In some embodiments, when all the data to be transmitted arrives at the storage area corresponding to the target address, the transmission process of the second processing card can generate another event identifier, such as event2. The data transmission client can periodically detect this event identifier; when it is valid, it can determine that the data requested by the application process has been transmitted and return a transmission completion message to the application process. The application process can then perform other processing operations based on the returned information, such as initiating an inter-card data transmission request for the next segment of data.

[0068] Figure 8 This is an exemplary block diagram of a data transmission system between multiple processing cards according to some embodiments of this specification. Figure 8 As shown, system 800 may include a data transmission server 810, one or more data transmission clients (such as data transmission clients 821, 822, etc.) and one or more transmission modules (such as transmission modules 831, 832, etc.).

[0069] The data transmission client is used to be called by application processes (such as application process 1, application process 2, etc.) to initiate inter-card data transmission requests to the data transmission server 810; the inter-card data transmission request includes a target address and a source address, and the target address and source address correspond to different processing cards.

[0070] The data transmission server 810 is used to respond to inter-card data transmission requests, select one or more target channels from two or more channels between the first processing card corresponding to the source address and the second processing card corresponding to the target address, and return the target channel identifier to the data transmission client.

[0071] The data transmission client is also used to obtain one or more data subsets of the data to be transmitted based on the number of target channels and the data to be transmitted; and to initiate data transmission instructions to the transmission module on the processing card associated with the target channel based on the one or more data subsets.

[0072] The transmission module is used to respond to data transmission instructions, transmit the corresponding data subset, and then transmit the data to be transmitted to the target address.

[0073] In some embodiments, the multiprocessor cards are located on the motherboard of the same processing device and include a CPU board and a GPU board; wherein, the number of application processes is more than one, and they are located on the CPU board and / or the GPU board respectively; the data transmission server is located on the CPU board; and the number of transmission modules is more than one, and they are located on the CPU board and / or the GPU board respectively.

[0074] For more information on each module, please refer to [link / reference]. Figure 2 The relevant explanations will not be repeated here. It should be understood that... Figure 8 The systems and modules shown can be implemented in various ways. For example, in some embodiments, the systems and modules can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated hardware. In some embodiments, the modules described above can be implemented by computer code, whereby, when the computer code is executed, the client can manifest as a function body and its interface, and the server and transmission module can manifest as different processes. Those skilled in the art will understand that the methods and systems described above can be implemented using computer-executable instructions and / or included in processor control code, for example, on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The systems and modules of this specification can be implemented not only by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., but also by software, for example, executed by various types of processors, or by a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0075] It should be noted that the above description of the system and its modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the modules, or construct subsystems connected to other modules, without departing from these principles. Alternatively, some modules may be split to obtain more modules or multiple units under a single module. Such modifications are all within the scope of this specification.

[0076] In some embodiments, System 800 can be encapsulated as a library for use by other program compilation systems. For example, it can be called by machine learning model frameworks (such as PyTorch, TensorFlow, etc.). Machine learning model frameworks are used for machine model training or prediction. Developers implement model training or prediction functions by writing corresponding applications. After the application is compiled by the framework, executable code is obtained. The executable code can then call modules in System 800 to achieve the desired results. Figure 2 The steps for data transfer between cards are shown.

[0077] Some embodiments of this specification also provide a method for data transmission between multiple processing cards, wherein the multiple processing cards have direct connection channels and / or indirect channels; the method includes: receiving an inter-card data transmission request; the inter-card data transmission request includes a target address, a source address, and an identifier of data to be transmitted, wherein the target address and the source address correspond to different processing cards; determining two or more channels between a first processing card corresponding to the source address and a second processing card corresponding to the target address; obtaining a weight value for each of the two or more channels; the weight value is positively correlated with the data transmission bandwidth of the channel and negatively correlated with the observation load of the channel; and selecting one or more target channels from the two or more channels based on the weight value for transmitting the data to be transmitted.

[0078] The data transfer method between multiple processing cards provided in some embodiments of this specification can be used not only in multiprocessor devices but also in computing clusters containing multiple computing devices. In this case, each computing device in the cluster can correspond to one processing card. This method can optimize data transfer in the computing cluster. For more details on the data transfer method between multiple processing cards, please refer to... Figure 3 The relevant explanations will not be repeated here.

[0079] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: (1) By optimizing the target channel selection strategy, the data transmission rate between processing cards is effectively accelerated: Experiments show that after adopting the data transmission method provided in the embodiments of this specification, the data transmission rate is increased by 2.8 times compared to data transmission from CPU card to GPU card through a single direct connection channel, and the data transmission rate is increased by 4.5 times compared to data transmission from GPU card through a single direct connection channel; (2) By optimizing the target channel selection strategy, the load on the channels between processing cards is effectively balanced, and the configuration of computing resources is optimized; (3) The data transmission method provided in the embodiments of this specification can be transparent to the application layer. Existing applications can also implement the card data transmission method provided in some embodiments of this specification by calling the interfaces or modules provided in this specification during the compilation stage without modification.

[0080] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0081] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0082] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0083] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0084] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0085] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0086] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A data transmission method between multiple processing cards, wherein the multiple processing cards have direct connection channels and indirect channels, and the indirect channels include direct connection channels with two or more hops; The multiprocessor cards are located on the motherboard of the same processing device; the method includes: Receive inter-card data transmission request; the inter-card data transmission request includes a target address and a source address, and the target address and source address correspond to different processing cards; Identify two or more channels between the first processing card corresponding to the source address and the second processing card corresponding to the target address; Select one or more target channels from the two or more channels to transmit the data to be transmitted; Based on the number of target channels and the data to be transmitted, one or more data subsets of the data to be transmitted are obtained, and the one or more data subsets are transmitted through the one or more target channels; Based on one or more data subsets, a data transmission command is initiated to the processing card associated with the target channel. The corresponding data subset is then transmitted through the processing card associated with the target channel, thereby transmitting the data to be transmitted to the target address. When the target channel is an indirect channel, it transmits a subset of data. The relay processing card is configured to read the subset of data from the storage area corresponding to the previous processing card and cache it locally, and then initiate a data transmission command to the next processing card. The relay processing card is a processing card in the indirect channel other than the first processing card and the second processing card. The relay processing card is further configured to cache the subset of data in a global storage area or a temporary storage area of ​​the transmission process through its transmission process: when the idle amount of the temporary storage area is greater than a set threshold, the subset of data is cached in the temporary storage area; otherwise, the subset of data is cached in the global storage area. The global storage area is a storage area on the relay processing card that is different from the temporary storage area.

2. The method as described in claim 1, further comprising that each of the more than one data subset has a different data volume, and the data subset with a larger data volume is transmitted through a channel with a larger data transmission bandwidth.

3. The method as described in claim 1, wherein the data to be transmitted includes the calculation results obtained by the application process.

4. The method of claim 1, wherein, in order to select one or more target channels from the two or more channels, the method further comprises: Obtain the component table; the component table includes the data transmission bandwidth of the direct connection channel between the multiprocessor cards; The minimum data transmission bandwidth among two or more direct-connected channels in an indirect channel is taken as the data transmission bandwidth of that indirect channel. A first weight value is determined for each of the two or more channels based on the data transmission bandwidth; the first weight value is positively correlated with the data transmission bandwidth of the channel. Obtain the load table; the load table includes the observed load of the direct connection channels between the multiprocessor cards; The observed load reflects the amount of data being transmitted and / or to be transmitted on the corresponding channel; The observation load of an indirect channel is obtained by summing the observation loads of two or more directly connected channels in the indirect channel. The second weight value of each of the two or more channels is determined based on the observed load. The second weight value is negatively correlated with the observation load of the channel; For each of the two or more channels: a weight value is obtained based on the first weight value and the second weight value, and one or more target channels are selected from the two or more channels based on the weight value.

5. The method as described in claim 1, wherein the data subset corresponds one-to-one with the target channel; the inter-card data transmission request further includes the offset of the storage area of ​​the data to be transmitted relative to the source address; The method further includes: Based on the offset, the data to be transmitted is split into data subsets with different data sizes; the data subset corresponding to the target channel with a larger data transmission bandwidth has a larger data size. Based on the source address and the amount of data in the data subset, the starting address and offset of each data subset are obtained; The data transmission instruction includes the starting address of the data subset and the subset offset.

6. The method as described in claim 1 or 5, wherein when the target channel is an indirect channel, a data transmission command is initiated by the data transmission client to the transmission process on the processing card associated with the target channel, comprising: The data transmission client initiates a data transmission command to the transmission process on the nearest processing card on the target channel, transmitting the corresponding data subset to the nearest processing card; The step of transmitting a corresponding data subset via a processing card associated with the target channel, and then transmitting the data to be transmitted to the target address, includes: The transmission process of the nearest processing card initiates a data transmission instruction to the transmission process of the next processing card on the target channel, transmitting the data subset to the next processing card, and so on, until the data subset is transmitted to the target address.

7. The method as described in claim 1, wherein the transmission process of the relay processing card is further configured to: Two or more buffer areas of preset size and two or more tasks are opened in the global storage area; the task is used to read data matching the preset size from the remaining data of the corresponding data subset according to the starting address of the data subset, the subset offset and the cached data subset in the data transmission instruction, and into the corresponding buffer area. in, The two or more tasks are executed in parallel or at a certain time interval.

8. The method of claim 7, wherein in the two or more tasks, after one task reads data into the corresponding buffer area, another task begins execution.

9. A storage medium for storing computer instructions that, when at least a portion of the computer instructions are executed by a processor or a processing card, implement the method as described in any one of claims 1 to 8.

10. A computer device comprising a plurality of processing cards and a storage medium, wherein the plurality of processing cards have direct connection channels and indirect channels, the indirect channels including two or more direct connection channels, the storage medium storing computer instructions, and the plurality of processing cards being configured to execute the computer instructions to implement the method as described in any one of claims 1 to 8.

Images