Multi-thread-based data transceiving method and device, medium and equipment

Abstract

The invention discloses a multi-thread-based data receiving and transmitting method and device, a medium and equipment. The method comprises the following steps: distributing a plurality of tasks to a plurality of processes of an OpenMPI-based open message transmission interface; labels are distributed to the multiple tasks respectively; sending corresponding relation information of the task, the process and the label to a task receiver; according to the corresponding relation information of the tasks and the labels, based on a multi-processing multi-thread mode opened by the OpenMP, the content sizes of the multiple tasks are sent to all the task receiving parties, so that cache regions are set by corresponding processes in all the task receiving parties according to the received content sizes; according to the corresponding relation information of the tasks and the labels, based on an OpenMP multi-thread mode, the contents of the multiple tasks are sent to all the task receiving parties, so that the corresponding processes in all the task receiving parties receive the tasks according to the labels and store the tasks in the corresponding cache regions. According to the invention, the security of data receiving and transmitting based on multiple threads can be improved.

Description

Technical Field

[0001] The present disclosure relates to data transmission technology, and in particular to a multi-threaded data transceiving method, a multi-threaded data transceiving device, a storage medium, and an electronic device. Background Art

[0002] OpenMPI (OpenMessage Passing Interface) can achieve parallel computing by distributing tasks to multiple processes.

[0003] OpenMPI can be used in a multi-threaded environment. However, in a multi-threaded environment, due to the acceleration of threads, the order in which the same computing node receives data is often uncertain. Therefore, when processing this data, the computing node may encounter the problem of disordered order of received data, which causes the order of data reception to be inconsistent with the expected processing order. The computing node cannot accurately process the received data, which may cause the received data to be unable to be accurately written into the corresponding cache area, thereby causing a cache overflow error.

[0004] How to ensure the security of data transmission and reception in a multi-threaded environment is a technical issue worthy of attention. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a multi-threaded data transmission and reception method, device, storage medium and electronic device.

[0006] According to a first aspect of an embodiment of the present disclosure, a multi-threaded data receiving and sending method is provided, comprising: assigning a plurality of tasks to a plurality of processes based on an OpenMPI open message transmission interface; assigning labels to the plurality of tasks respectively, wherein different tasks have different labels; sending correspondence information of the tasks, processes and labels to task receivers, so that the processes in each task receiver wait to receive the corresponding tasks according to the labels; based on the multi-processing and multi-threading method open to OpenMP, sending the content size of the plurality of tasks to each task receiver respectively according to the correspondence information of the tasks and labels, so that the corresponding processes in each task receiver respectively set a cache area according to the received content size; based on the correspondence information of the tasks and labels, based on the OpenMP multi-threading method, sending the contents of the plurality of tasks to each task receiver respectively according to the label and storing them in the corresponding cache area.

[0007] According to a second aspect of an embodiment of the present disclosure, another multi-threaded data receiving and sending method is provided, including: receiving correspondence information of tasks, processes and labels from a task sender; according to the correspondence information, making the corresponding process based on OpenMPI in a waiting receiving state on the corresponding label; when the process receives the content size of the task from the task sender based on the OpenMP multi-threading method according to the label in the correspondence information, the process sets a buffer area according to the content size of the task; when the process receives the content of the task from the task sender based on the OpenMP multi-threading method according to the label in the correspondence information, the process stores the content of the task in the buffer area; the process executes the task according to the content of the task stored in the buffer area, and returns the execution result of the task to the task sender according to the label in the correspondence information.

[0008] According to a third aspect of an embodiment of the present disclosure, a multi-threaded data transceiver device is provided, comprising: a task allocation module, for allocating multiple tasks to multiple processes based on OpenMPI; a label allocation module, for respectively allocating labels to the multiple tasks, wherein different tasks have different labels; a first label handshake module, for sending the correspondence information of the tasks, processes and labels to the task receiver, so that the processes in each task receiver wait to receive the corresponding tasks according to the labels; a first sending module, for sending the content size of the multiple tasks to each task receiver based on the OpenMP multi-threading method according to the correspondence information of the tasks and labels, so that the corresponding processes in each task receiver respectively set a cache area according to the received content size; a second sending module, for sending the content of the multiple tasks to each task receiver based on the OpenMP multi-threading method according to the correspondence information of the tasks and labels, so that the corresponding processes in each task receiver respectively receive the tasks according to the labels and store them in the corresponding cache areas.

[0009] According to a fourth aspect of an embodiment of the present disclosure, a multi-threaded data transceiver device is provided, comprising: a second label handshake module, used to receive correspondence information of tasks, processes and labels from a task sender; a waiting reception module, used to enable a corresponding process based on OpenMPI to be in a waiting reception state on a corresponding label according to the correspondence information; a buffer area setting module, used to enable the process to set a buffer area according to the content size of the task when the process receives the content size of the task from the task sender based on the OpenMP multi-threading method according to the label in the correspondence information; a task storage module, used to enable the process to store the content of the task in the buffer area when the process receives the content of the task from the task sender based on the OpenMP multi-threading method according to the label in the correspondence information; a task execution module, used to enable the process to execute the task according to the content of the task stored in the buffer area, and enable the process to return the execution result of the task to the task sender according to the label in the correspondence information.

[0010] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to implement any of the above methods.

[0011] According to the sixth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement any of the above methods.

[0012] Based on the above embodiments of the present disclosure, a multi-threaded data transmission and reception method, device, storage medium and electronic device are provided. By assigning labels and processes to all tasks respectively, and sending the corresponding relationship information of tasks, processes and labels to each task receiver respectively, the task sender and the task receiver implement a label handshake for data transmission and reception, so that the task sender and each task receiver can use the label to adopt the OpenMP multi-threading method to realize the content size of the task and the transmission and reception of the content of the task. Since the process assigned to the task in the task receiver can adopt the OpenMP multi-threading method, according to the content size of the task from the task sender, the same size of the buffer area is set for the task quickly and accurately, therefore, the process in the task receiver can adopt the OpenMP multi-threading method to quickly and accurately store the content of the task from the task sender based on the label in the buffer area set for it, while ensuring the efficiency of data transmission and reception, the occurrence of the buffer overflow phenomenon is effectively avoided. It can be seen that the technical solution provided by the present disclosure is conducive to enriching the multi-threaded data transmission and reception technology, and is conducive to improving the data transmission and reception security of the system.

[0013] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] By describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 This is a schematic diagram of an application scenario of a multi-threaded data receiving and sending method disclosed in the present invention;

[0016] Figure 2 It is a flow chart of an embodiment of a multi-threaded data receiving and sending method disclosed in the present invention;

[0017] Figure 3 A flowchart of an embodiment of allocating multiple tasks to multiple processes according to the present disclosure;

[0018] Figure 4 It is a flow chart of an embodiment of sending corresponding relationship information to a task receiver according to the present disclosure;

[0019] Figure 5 It is a flowchart of another embodiment of the multi-threaded data receiving and sending method disclosed in the present invention;

[0020] Figure 6 A specific implementation flow chart of the multi-threaded data receiving and sending method disclosed in the present invention;

[0021] Figure 7 It is a structural diagram of an embodiment of a multi-threaded data transceiver device disclosed in the present invention;

[0022] Figure 8 It is a structural schematic diagram of another embodiment of the multi-threaded data transceiver device disclosed in the present invention;

[0023] Fig. 9 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The exemplary embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0025] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0026] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0027] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0028] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0029] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0030] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0031] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0032] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0033] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0034] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0035] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0036] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks can be performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0037] SUMMARY OF THE DISCLOSURE

[0038] In the process of realizing the present disclosure, the inventors found that in order to avoid the occurrence of the problem of buffer overflow caused by the order in which the receiver receives data in a multi-threaded environment that may not match the expected order, the solutions currently adopted are mostly based on certain strategies to ensure that the order in which the data is received is consistent with the expected order, so as to ensure that the data can be stored in the corresponding position of the buffer to avoid the phenomenon of buffer overflow. Such a solution can be considered as a solution to the root cause of the problem. However, some solutions will affect the data transceiving performance of the system (such as affecting the data transceiving efficiency of the system, etc.). In addition, the factors that cause the order in which the receiver receives data to be inconsistent with the expected order may not be single. On the basis of not affecting the data transceiving performance, shielding all possible factors to avoid the phenomenon of buffer overflow is challenging. If the phenomenon that the order in which the data is received may be inconsistent with the expected order exists, solving the phenomenon of buffer overflow from other angles is not only conducive to enriching the data transceiving technology based on multi-threading, but also conducive to avoiding the occurrence of buffer overflow caused by not shielding all possible factors (such as unknown factors that will appear in the future, etc.) on the basis of ensuring the data transceiving performance. That is to say, regardless of whether the order in which the receiver receives data matches the expected order, ensuring that the receiver can store the received data in the correct location of the buffer based on correct understanding can also solve the buffer overflow problem.

[0039] Exemplary Overview

[0040] The technical solution of multi-threaded data transmission and reception disclosed in the present invention can be applied to a variety of application scenarios. For example, the technical solution provided by the present invention can be applied to application scenarios such as file reading and writing, background task execution, cloud computing, and client-server models. In any application scenario, a specific example of implementing the technical solution of the present invention is as follows: Figure 1 shown.

[0041] Figure 1 In the example, the master node 100 (i.e., the task sender) is set to be connected to multiple computing nodes (i.e., the task receivers) such as computing node 101-1, computing node 101-2, ..., and computing node 101-n, where n is an integer greater than 2), and any two computing nodes can be located in two different hardware devices or in the same hardware device. The master node 100 can be located in two different hardware devices with any computing node, or in the same hardware device with some computing nodes.

[0042] OpenMPI is installed in the master node 100 and each computing node. The master node 100, computing node 101-1, computing node 101-2, ... and computing node 101-n form an OpenMPI cluster.

[0043] There are multiple tasks that need to be executed currently in the master node 100. The master node 100 uses a predetermined allocation strategy to allocate all the tasks that need to be executed currently to multiple processes in the OpenMPI cluster. The processes in the OpenMPI cluster include: processes in all computing nodes. The master node 100 allocates corresponding labels and processes to each task, and performs label handshake with each computing node that executes the task. After the label handshake, the corresponding process in the computing node is in a state of waiting to receive based on the label. Before sending the content of each task, the master node 100 can first send the content size of each task to the corresponding computing node based on the label corresponding to each task. After receiving the content size of the corresponding task based on the label held by it, the corresponding process in the computing node sets the corresponding cache area, and after receiving the content of the task based on the label held by it, the content of the task is stored in the cache area. After executing the corresponding task, the corresponding process in each computing node returns the execution result to the master node 100. The master node 100 waits to receive the execution results returned by each computing node according to the label corresponding to each task, and continues to perform subsequent operations according to the received calculation results.

[0044] Exemplary Methods

[0045] Figure 2 FIG. 1 is a flow chart of an embodiment of a multi-threaded data transmission and reception method disclosed in the present invention. Figure 2 The method shown is executed on the task sender (also called the task distributor), which is usually a master node based on OpenMPI. The method mainly includes: S200, S201, S202, S203 and S204. Figure 2 Each step in is described separately.

[0046] S200, allocating multiple tasks to multiple processes based on OpenMPI.

[0047] The task in the present disclosure refers to a basic unit of work that needs to be completed using computing resources in a multi-process environment. For example, a task may include one or more instruction sequences. All tasks in the present disclosure usually have their own index values. For example, all tasks are numbered one by one in sequence, and the sequential numbering of the tasks can be used as the index value of the task. The index value of the task can also be called the identifier of the task. The index value of the task is unique, that is, the index values ​​of any two tasks in the multiple tasks in S200 are not the same.

[0048] The index values ​​of all tasks in the present disclosure can be represented by the subscript or suffix of the task. For example, assuming that the number of the multiple tasks in S200 is N (N is an integer greater than 1), and the index values ​​of the N tasks are 0 to N-1, then the N tasks can be represented as task 0 、task 1 , ... and task N-1 , can also be expressed as task0, task1, ... and taskN-1.

[0049] The present disclosure can organize each task into a one-dimensional array, and the content of the one-dimensional array can be all the information contained in the task (such as all instructions, etc.). The one-dimensional arrays of all tasks can form a two-dimensional array, and in the subsequent description, the two-dimensional array can be named totalTask.

[0050] OpenMPI in this disclosure refers to a technology that implements parallel computing in a distributed memory system and allows processes on different computing nodes to communicate and coordinate by message passing. The OpenMPI-based process in this disclosure refers to a process in an OpenMPI cluster, and a process in an OpenMPI cluster can be considered to be a process in a process pool formed by resources in multiple OpenMPI-based nodes (such as all nodes forming an OpenMPI cluster).

[0051] In one example, when the number of processes in the OpenMPI cluster is not less than the number of tasks, the present disclosure can assign one task to one process, and different tasks are assigned to different processes, that is, there is a one-to-one relationship between processes and tasks.

[0052] In another example, when the number of processes in the OpenMPI cluster is less than the number of tasks, the present disclosure may also assign two or more tasks to a process, that is, different tasks may be assigned to the same process, that is, there may be a one-to-many relationship between processes and tasks.

[0053] In one example, the present disclosure may use multiple strategies to implement task allocation, for example, an average allocation strategy may be used to evenly allocate multiple tasks to processes in an OpenMPI cluster. For example, when an OpenMPI cluster contains multiple processes, and the number of processes contained is not less than the number of tasks currently to be allocated, the present disclosure uses an average allocation strategy to allocate multiple tasks to multiple different processes in the OpenMPI cluster. Since the processes allocated by the task sender to any two tasks are different, the task receiver only receives and executes one task through one process.

[0054] The present disclosure adopts an average distribution strategy for task distribution, which is conducive to avoiding the occurrence of process load imbalance in the OpenMPI cluster and making balanced use of resources of all nodes (such as all computing nodes) in the OpenMPI cluster, thereby helping to avoid the degradation of system performance of the OpenMPI cluster and the waste of computing resources due to low utilization.

[0055] A specific example of distributing multiple tasks to multiple different processes based on OpenMPI is as follows: Figure 3 shown.

[0056] Figure 3 In S300, obtain the total number of processes in the OpenMPI cluster.

[0057] All processes in the OpenMPI cluster in the present disclosure generally have respective index values. In one example, the present disclosure may sequentially number all processes in the OpenMPI cluster one by one, so that each process has a sequential number, and the present disclosure may use the sequential number of the process as the index value of the process. The index value of the process may also be referred to as the identifier of the process. The index value of the process is unique, that is, the index values ​​of any two processes in the multiple processes in S200 are not the same, and the index values ​​of any two processes in all processes in the OpenMPI cluster are not the same.

[0058] The index values ​​of all processes in the present disclosure can be represented by the subscript or suffix of the process. For example, if an OpenMPI cluster includes M (M is an integer greater than 1) processes, and the index values ​​of these M processes are 0 to M-1, then these M processes can be represented by rank 0 , rank 1 , ... and rank M-1 , can also be expressed as rank0, rank1, ... and rankM-1. Optionally, the present disclosure can obtain the total number of processes in the OpenMPI cluster by calling the corresponding function in OpenMPI. The present disclosure does not limit the specific implementation method of obtaining the total number of processes.

[0059] S301, based on the OpenMP (Open Multi-Processing) multi-threading method, calculate the index value of each task and perform modulo processing on the difference between the total number of processes and a first predetermined value to obtain multiple remainders.

[0060] The operation of performing modulo processing on all tasks separately in the present disclosure can be executed in parallel based on OpenMP multithreading. OpenMP in the present disclosure refers to a technology that can implement parallel computing on a multi-core processor in a single computer and accelerate computing by sharing memory between multiple threads in the same program.

[0061] As a specific example, the present disclosure can create threads according to the number of tasks so that the number of threads is the same as the number of tasks, and one thread performs a modulo processing operation for one task. S301 of the present disclosure can also be described as: for any task, a thread is used to perform modulo processing on the index value of the task to the difference between the total number of processes and a first predetermined value (a non-zero positive integer), thereby obtaining a remainder, and the task corresponds to the remainder. Since there is a process in the OpenMPI cluster, the index value of the process is the remainder, so the task in the present disclosure can correspond to the only process in the OpenMPI cluster through the remainder. The first predetermined value in the present disclosure can be set according to actual needs, such as the first predetermined value can be 1.

[0062] In one example, it is assumed that there are N tasks that need to be assigned processes, and the total number of processes obtained in S300 is M. If the N tasks are represented as task 0 、task 1 , ... and task N-1 , and denote the M processes as ranks 0 , rank 1 , ... and rank M-1 , then for task task i (i is an integer greater than or equal to 0 and less than N), for task task i The remainder processing performed can be expressed by the following formula (1):

[0063] X=i%(mpiSize-1) Formula (1)

[0064] In the above formula (1), the first predetermined value is 1, mpiSize is equal to the total number of processes M in the OpenMPI cluster, i is the index value of the task, and X is the number of processes for the task task. i Calculate the remainder obtained.

[0065] The present disclosure can utilize N threads to execute the calculation operation of the above formula (1), so that the index value of the process assigned to each task can be obtained in parallel.

[0066] S302: Allocate each task to a process whose index value is the corresponding remainder.

[0067] Since each task corresponds to a remainder, and the remainder is usually the index value of a process, the remainder can be used to assign a task to a process. For example, according to the above formula (1), the i-th task is assigned to the process with index value X, that is, rank x .

[0068] The present disclosure uses the first predetermined value to perform remainder processing and uses the remainder to determine the process corresponding to the task, which is not only conducive to evenly distributing all tasks to processes in the OpenMPI cluster to make the process load as balanced as possible, but also can avoid the phenomenon that the task sender sends the task to itself.

[0069] S201, assigning labels to multiple tasks respectively.

[0070] The present disclosure assigns a label to each task, and different tasks are assigned different labels, that is, the label of a task is unique in one execution process of the data transmission and reception method of the present disclosure. The functions of the label in the present disclosure include: enabling the task sender and the task receiver to perform a handshake operation based on the label, and the task sender and the task receiver should communicate on the pre-defined label, so that both parties can achieve accurate data transmission and reception processing.

[0071] In one example, the operation of assigning labels to all tasks in the present disclosure can be executed in parallel based on OpenMP multi-threading, so that the labels assigned to each task can be obtained in parallel.

[0072] In one example, for any task, the present disclosure can assign two tags to the task, one tag corresponding to the content size of the task, and one tag corresponding to the content of the task. That is, the present disclosure assigns a first tag to the content size of the task, and assigns a second tag to the content of the task. The first tag has the function of causing the corresponding process of the task receiver to wait for the content size of the task based on the first tag, and the second tag has the function of causing the corresponding process of the task receiver to wait for the content of the task based on the second tag.

[0073] Since the labels of the tasks in the present disclosure are unique, during one execution of the data sending and receiving method of the present disclosure, the first labels of any two tasks are different, the second labels of any two tasks are also different, the first label and the second label of the same task are different, and the first label of any task and the second label of any task are also different.

[0074] In one example, the first label and the second label assigned to a task may be two labels that are sequentially continuous, and the first labels and the second labels assigned to all tasks may form a numerically continuous label sequence.

[0075] Optionally, the present disclosure can allocate the first label and the second label to all tasks respectively based on the OpenMP multi-threading method. For example, the present disclosure can create threads according to the number of tasks, so that the number of threads is the same as the number of tasks, and one thread calculates the first label and the second label corresponding to one task. In a specific example, the present disclosure can use the OpenMP for instruction to perform multi-threaded acceleration on the label allocation operation of all tasks, parallelize the label allocation operation of all tasks, and realize fast label allocation, which is conducive to improving the efficiency of data transmission and reception.

[0076] The present disclosure adopts the OpenMP multi-threading method to assign a unique first label and a second label to a task, so that the task receiver can clearly know the content size of the task based on the first label, so that a cache area of ​​the same size can be quickly and accurately set for the task, and then the task receiver can quickly and accurately store the content of the task received based on the second label in the cache area, while maximizing the efficiency of data transmission and reception, and effectively avoiding the occurrence of cache overflow.

[0077] Optionally, the process of assigning the first label and the second label to a task disclosed in the present invention may be as follows: for any task, the present invention may perform calculations based on the index value of the task and the preset label starting value, thereby obtaining the first label of the task, and then use a label that is sequentially connected to the first label as the second label of the task. There are many ways to calculate the index value of a task and the preset label starting value, which can be set according to actual conditions. A specific example: for any task, obtain the product of the index value of the task and the second preset value, use the sum of the preset label starting value and the above product as the first label of the task, and use the sum of the first label and the third preset value as the second label of the task. The preset label starting value is usually a positive integer, the second preset value is usually a non-zero positive integer, and the third preset value is usually a non-zero positive integer less than the second preset value.

[0078] In one example, for any task task i (i is an integer greater than or equal to 0 and less than N, and N is the number of tasks), the label allocation process performed on the task can be expressed by the following formula (2) and formula (3):

[0079] Tag1=TAGSTART+i*2 Formula (2)

[0080] Tag2=TAGSTART+i*2+1 Formula (3)

[0081] Among them, tag1 represents the task task i The first tag assigned, tag2 represents the task task iThe second tag assigned, TAGSTART is the preset tag start value, the second predetermined value is 2, and the third predetermined value is 1.

[0082] By introducing the task index value and the preset label starting value in the process of calculating the first label and the second label of the task, it is helpful to avoid the phenomenon that the first label and the second label set for the task conflict with the labels in other programs, thereby improving the system security of the OpenMPI cluster.

[0083] In one example, the present disclosure is based on the OpenMP multi-threading method, and the pseudo code for implementing S200 and S201 can be expressed as follows:

[0084]

[0085] In the above pseudo code, TaskList represents a task list set for all tasks, and the task list includes at least: the index value of the task, the index value of the process assigned to the task, and the label assigned to the task, where the label can include at least one of the first label and the second label. Since the first label and the second label of the task have a certain association relationship (such as the second label is 1 plus the first label), even if the task list only contains one of the first label and the second label, in the subsequent process of using the label, the present disclosure can obtain the other label of the task by deducing calculation.

[0086] In addition, the present disclosure does not exclude the situation where a label is assigned to each task, that is, for any task, the task sender uses the same label when sending the content size of the task and the content of the task. In this case, the task receiver can use the data received for the first time based on the label as the content size of the task, and the data received for the second time based on the label as the content of the task.

[0087] S202: Sending the correspondence information of the task, process and label to the task receiver, so that the corresponding process in each task receiver waits to receive the corresponding task according to the label.

[0088] Each task in the present disclosure corresponds to a corresponding relationship information, and the corresponding relationship information corresponding to any two tasks is different. For example, each task corresponds to a row in the task list TaskList. A corresponding relationship information can be expressed in the form of a one-dimensional array, and the one-dimensional array may include: the index value of the task, the index value of the process, and the label (such as at least one of the first label and the second label). In an example, the present disclosure can use the content of each row in the task list TaskList to form a corresponding relationship information respectively. This step can realize the label handshake of each task between the task sender and the task receiver by sending the corresponding relationship information of the task, process and label to the task receiver.

[0089] For any task, the process to which the task sender sends the correspondence information of the task is usually different from the process assigned to the task. In one example, the index value of the process to which the task sender sends the correspondence information of the task can have a certain correlation with the index value of the task. That is to say, for any task, the present disclosure can calculate the index value of the process that processes the correspondence information of the task based on the index value of the task, and then perform the corresponding relationship information sending operation corresponding to the task according to the process with the calculated index value, so as to realize the label handshake of both parties.

[0090] The present disclosure can calculate the index value of the process to which the corresponding relationship information of each task is sent based on the OpenMP multi-threading method, and perform the operation of sending the corresponding relationship information of each task based on the index value based on the OpenMP multi-threading method. In an example, the present disclosure can use loop nesting technology (collapse instruction) to implement step S202 to improve the execution speed of the label handshake.

[0091] A specific example of sending the corresponding relationship information of tasks, processes and tags to the task receiver in the present disclosure is as follows: Figure 4 shown.

[0092] Figure 4 In S400, the index value of each task, the index value of the process corresponding to each task, and the first label and the second label respectively assigned to each task are respectively formed into a one-dimensional array.

[0093] The number of one-dimensional arrays formed in the present disclosure is the same as the number of tasks, and a one-dimensional array can be represented as (task i , rank j , tag k ), where the tag kIn other embodiments, the task sender and the task receiver agree in advance that the above tags k Only one of the first label and the second label may be included, and the task receiver may calculate and obtain the other label according to a label in the received one-dimensional array. In addition, the present disclosure may read the content in each row in TaskList based on the OpenMP multithreading method, thereby obtaining the one-dimensional array of all tasks in parallel.

[0094] S401 . Determine the index value of the tag handshake process corresponding to each task according to the index value of each task.

[0095] The tag handshake process in the present disclosure refers to a process for implementing a tag handshake between a task sender and a task receiver, that is, a process for receiving and processing the correspondence information between tasks, processes and tags.

[0096] For any task, the tag handshake process corresponding to the task is usually different from the process assigned to the task, and the index value of the tag handshake process corresponding to the task is usually associated with the index value of the task. For example, for any task, the index value of the tag handshake process corresponding to the task is the sum of the index value of the task and a predetermined value (a non-zero positive integer, such as 1, etc.).

[0097] S402 , based on the OpenMP multi-threading method, sending the one-dimensional array corresponding to each task according to the index value of each label handshake process.

[0098] As a specific example, for any task (such as task i ), the task sender sends the task (such as task i ) index value (such as i), the task (such as task i) and the first label and the second label assigned to the task (labels are used to identify tasks in inter-process communication) are sent to the process (such as ranki+1, that is, the label handshake process) whose index value is the sum of the index value of the task (such as i) and a predetermined value (a non-zero positive integer, such as 1, etc.). The label handshake process can be implemented based on inter-process communication, that is, the index value of the task and the two labels corresponding to the task are sent to the corresponding process through inter-process communication, so that the corresponding process in the task receiver (i.e., the process assigned to the task) waits to receive the relevant information of the task (such as the size of the task content, the content of the task, etc.) according to the first label and the second label in the corresponding relationship information. For example, after the process ranki+1 receives the one-dimensional array, the index value (such as i) of the task (such as taski) and the first label and the second label assigned to the task can be notified to the process assigned to the task in the corresponding relationship through inter-process communication, so that the process assigned to the task can know the index value of the task it needs to receive and the first label and the second label corresponding to the task, and then the process assigned to the task can wait to receive the content size of the task based on the first label, and wait to receive the content of the task based on the second label. By using the OpenMP multi-threading method and inter-process communication technology in the label handshake process, it is conducive to quickly and accurately implement the label handshake process.

[0099] S203 , according to the correspondence information between the task and the label, based on the OpenMP multithreading method, sending the content size of the task to each task receiver respectively, so that the corresponding process in each task receiver respectively sets the buffer area according to the received content size.

[0100] The content size of the task in the present disclosure can be considered as the size of the cache space required by the task receiver when storing the content of the received task in the cache area. For any task, the task sender of the present disclosure can generate a message based on the first tag of the task and the content size of the task, and send the message to the task receiver. Since the corresponding process in the receiver of the task is in a waiting state based on the first tag and the second tag, the process can receive the message based on the first tag and obtain the content size of the task in the message, so that the process can set a cache area with the same content size as the task according to the content size of the task it obtains. The cache area can be considered as the cache area corresponding to the first tag.

[0101] In one example, the present disclosure may use an asynchronous sending method based on OpenMP multithreading to send the content size of each task. The asynchronous sending method based on OpenMP multithreading may refer to the task sender using the relevant instructions of OpenMP (such as the for instruction) to send multithreaded data, and the task sender does not need to block and wait for the sending result. Specifically, for any task, a thread in the task sender can send the message generated by the content size of the task and the first tag of the task to the receiver of the task based on the asynchronous sending method of OpenMP multithreading according to the corresponding relationship information between the task and the first tag. The present disclosure can create threads for all tasks respectively, one thread corresponds to one task, and for any task, the thread generates a message carrying the first tag corresponding to the task and the content size of the task for the task, and sends the message to the task receiver, and the threads corresponding to all tasks can perform their respective operations in parallel. After sending the content size of each task, the task sender can wait to receive the buffer allocation execution results returned by each task receiver respectively based on the first tag of each task held by it. Since the processes in each task receiver in the present disclosure are all based on the content size of the task received by the first label, the present disclosure does not need to pay attention to the order in which the content size of the tasks is received by the task receivers. By adopting the OpenMP multi-threaded asynchronous sending method to send the content size of each task, it is beneficial to achieve rapid distribution of the content size of each task, thereby ensuring data sending and receiving efficiency.

[0102] S204, according to the correspondence information between the tasks and the labels, based on the OpenMP multi-threading method, the contents of the multiple tasks are respectively sent to each task receiver, so that the corresponding process in each task receiver respectively receives the tasks according to the labels and stores them in the corresponding buffer area.

[0103] The content of the task in the present disclosure can be considered as the task itself, for example, all instructions contained in the task, etc. For any task, the task sender of the present disclosure can generate a message based on the second tag of the task and the content of the task, and send the message to the task receiver. Since the corresponding process in the task receiver is in a waiting state based on the first tag and the second tag, the process can receive the message based on the second tag and obtain the content of the task carried in the message, so that the process can store the obtained task content in the cache area pre-set based on the first tag.

[0104] In one example, the present disclosure can use an asynchronous sending method based on OpenMP multithreading to send the content of each task. Specifically, for any task, a thread in the task sender can send a message generated by using the content of the task and the second tag of the task to the receiver of the task based on the asynchronous sending method of OpenMP multithreading according to the corresponding relationship information between the task and the second tag. The present disclosure can create threads for all tasks respectively, one thread corresponds to one task, and for any task, the thread generates a message carrying the second tag corresponding to the task and the content of the task for the task, and sends the message to the task receiver, and the threads corresponding to all tasks can execute their respective operations in parallel. After sending the content of each task, the task sender can wait to receive the task execution results returned by each task receiver based on the second tag of each task it holds. In addition, the process used by the sender to send the content of the task can be determined according to the index value of the task. For example, the index value of the process that sends the content of the i-th task can be the sum of i and a predetermined value (such as 1). Since the processes in each task receiver in the present disclosure are all based on the content of the task received by the second label, the present disclosure does not need to pay attention to the order in which the content of the task is received by the task receiver. By adopting the OpenMP multi-threaded asynchronous sending method to send the content of each task, it is beneficial to achieve rapid distribution of the content of each task, thereby ensuring data sending and receiving efficiency.

[0105] In one example, the present disclosure is based on the asynchronous sending mode of OpenMP multi-threading and adopts loop nesting technology. The pseudo code for implementing step S204 can be expressed as follows:

[0106]

[0107]

[0108] The disclosure allocates labels and processes to all tasks respectively, and sends the corresponding relationship information of tasks, processes and labels to each task receiver respectively, so that the task sender and the task receiver can realize the label handshake for data transmission and reception, so that the task sender and each task receiver can use the label and adopt the OpenMP multi-threading method to realize the content size of the task and the transmission and reception of the content of the task. Since the process allocated to the task in the task receiver can adopt the OpenMP multi-threading method, according to the content size of the task from the task sender, the same size of the buffer area can be quickly and accurately set for the task, therefore, the process in the task receiver can adopt the OpenMP multi-threading method, the content of the task from the task sender received based on the label is quickly and accurately stored in the buffer area set for it, while ensuring the efficiency of data transmission and reception, effectively avoiding the occurrence of the buffer overflow phenomenon. It can be seen that the technical solution provided by the disclosure is conducive to enriching the data transmission and reception technology based on multi-threading, and is conducive to improving the data transmission and reception security of the system.

[0109] Figure 5 FIG. 1 is a flow chart of another embodiment of the multi-threaded data receiving and sending method disclosed in the present invention. Figure 5 The method shown is executed on a task receiver (also referred to as a task executor), which is usually a computing node based on OpenMPI. The method mainly includes: S500, S501, S502, S503 and S504. Figure 5 Each step in is described separately.

[0110] S500: Receive the correspondence information of the task, process and label from the task sender.

[0111] The process in this step is the process assigned to the task by the task sender, and the label in this step is the label assigned to the task by the task sender. The correspondence information in this step can be sent to the label handshake process by the task sender, and provided to the task receiver by the label handshake process based on inter-process communication. The correspondence information is used to indicate that the corresponding process in the task receiver should wait to receive the content size and content of a task based on the corresponding label. The relationship between the index value of the task, the index value of the process and the label in the correspondence information can be found in the above description. Figure 2 The description of the relevant steps in will not be described in detail here.

[0112] S501 . According to the above correspondence information, a corresponding process based on OpenMPI is placed in a waiting-for-receiving state on a corresponding label.

[0113] Specifically, the corresponding process based on OpenMPI in this step refers to the process assigned by the task sender to the task that the task receiver is responsible for. After the label handshake process is notified, the process can wait to receive the content size and content of the task sent by the task sender based on the first label and the second label it holds.

[0114] S502: When the process receives the content size of the task from the task sender based on the OpenMP multi-threading method according to the label in the corresponding relationship information, it sets the buffer area according to the content size of the task.

[0115] Optionally, when a process receives a message carrying the first tag and the content size of the task according to the first tag it holds, it can obtain the content size of the task from the message and perform a buffer setting operation, and the size of the buffer set by the process is usually consistent with the content size of the task. The buffer can be considered as the buffer corresponding to the first tag / second tag. After the process successfully performs the buffer setting operation, the buffer setting result can be returned to the task sender according to the first tag, such as carrying the first tag and the buffer setting result in a response message, and sending the response message to the task sender. The task sender waits to receive the buffer setting result returned by each task receiver based on the first tag of each task it holds.

[0116] S503 , when the process receives the content of the task from the task sender based on the OpenMP multi-threading method according to the label in the corresponding relationship information, it stores the content of the task in the buffer area.

[0117] Optionally, when a process receives a message carrying a second tag and the content size of a task according to the second tag in the corresponding relationship information it holds, it can obtain the content of the task from the message and store the content of the task in a cache area corresponding to the first tag / second tag.

[0118] S504: The process executes the task according to the content of the task stored in the above-mentioned cache area, and returns the execution result of the task to the task sender according to the label in the above-mentioned corresponding relationship information.

[0119] Optionally, after executing the task according to the content of the task stored in the cache, the process can return the task execution result to the task sender according to the second tag assigned to the task, such as carrying the second tag and the task execution result in a response message, and sending the response message to the task sender. The task sender waits to receive the task execution result returned by each task receiver based on the second tag of each task it holds.

[0120] The pseudo code of an example of data sending and receiving operations performed by all task receivers of the present disclosure can be expressed as follows:

[0121]

[0122] The task receiver in the present disclosure performs a label handshake with the task sender by using the corresponding relationship information of the task, process and label, so that the task receiver can use the label to wait for the content size and content of the task assigned to it, and perform the receiving operation in an OpenMP multi-threaded manner, which is conducive to improving the content size of the task and the receiving speed of the content of the task; because the process in the task receiver is a cache area set based on the content size of the task corresponding to the label, no matter whether the content order of the task received by the task receiver is consistent with the expected order, it can be accurately stored by the task receiver in the cache area corresponding to the label, thereby avoiding the occurrence of the buffer overflow problem. It can be seen that the technical solution provided by the present disclosure is conducive to enriching the data receiving and sending technology based on multi-threading, and is conducive to improving the data receiving and sending security of the system.

[0123] A specific implementation process of the multi-threaded data transmission and reception method disclosed in the present invention can be as follows: Figure 6 shown.

[0124] Figure 6 In step S600, the steps of starting this process are as follows.

[0125] S601, the master node (also called the OpenMPI master node) organizes each task to be sent into a one-dimensional array, and the content of a one-dimensional array can be all the information contained in a task. The one-dimensional arrays of all tasks to be sent can form a two-dimensional array totalTask.

[0126] S602, the master node assigns processes and labels to all tasks to be sent. For example, the master node adopts an average distribution strategy to evenly distribute all tasks to be sent to processes in the OpenMPI cluster. Usually, the processes assigned by the master node to any two tasks are different. For another example, the master node uses the index value of each task and the preset label starting value to calculate and assign the first label and the second label to all tasks.

[0127] S603, the master node performs label handshake with each computing node respectively. For example, the master node starts OpenMP multithreading, calculates the index value of the process to which the corresponding relationship information corresponding to each task is sent, and uses OpenMP multithreading to perform the operation of sending the corresponding relationship information corresponding to each task according to the obtained index values ​​to implement label handshake.

[0128] S604, the corresponding processes in each computing node wait to receive data on the designated tags. The corresponding processes here refer to the processes assigned by the master node to the tasks. The designated tags here are the tags in the tag handshake operation in the above steps, and the designated tags may include the first tag and the second tag. The data here may be in the form of a message, and may specifically be a message carrying the first tag and the content size of the task, or a message carrying the second tag and the content of the task, etc.

[0129] S605 . When each computing node detects the arrival of a designated label (such as the first label or the second label), it starts the OpenMP multithreading, such as by using the OpenMP for instruction.

[0130] S606, each computing node receives the content size of the task on the specified tag (such as the first tag) using OpenMP multithreading, sets a buffer area according to the content size of the task, and returns the result of setting the buffer area to the master node based on the specified tag (such as the first tag). Each computing node receives the content of the task on the specified tag (such as the second tag) using OpenMP multithreading, stores the content of the task in the buffer area set based on the first tag, executes the instructions contained in the task, and returns the task execution result to the master node based on the specified tag (such as the second tag).

[0131] S607. The master node uses OpenMP multithreading (such as starting OpenMP multithreading by using the OpenMP for instruction) to send data (such as the content size of each task and the content of each task) to the computing nodes based on the specified tags of all tasks (such as the first tag of each task and the second tag of each task), and waits to receive the results returned by each computing node (such as the cache setting result and the task execution result) on the specified tags (such as the first tag of each task and the second tag of each task).

[0132] S608. When the master node detects that a designated label (such as the first label or the second label) has arrived, the master node starts the OpenMP multithreading, such as by using the OpenMP for instruction to start the OpenMP multithreading.

[0133] S609: The master node uses OpenMP multithreading to receive results returned by each computing node on a specified label (such as the first label or the second label of all tasks), such as a cache setting result and a task execution result.

[0134] S610, end the steps of this process.

[0135] Exemplary Devices

[0136] Figure 7FIG. 1 is a schematic diagram of a structure of an embodiment of a multi-threaded data transceiver device disclosed in the present invention. The device of this embodiment can be used to implement the corresponding method embodiment of the present invention. Figure 7 The device shown includes: a task allocation module 700 , a label allocation module 701 , a first label handshake module 702 , a first sending module 703 and a second sending module 704 .

[0137] The task allocation module 700 is mainly used to allocate multiple tasks to multiple processes based on OpenMPI. Optionally, the task allocation module 700 can allocate multiple tasks to multiple different processes in the OpenMPI cluster; wherein the processes allocated to any two tasks are different. In one example, the task allocation module 700 includes: a first submodule 7001, a second submodule 7002 and a third submodule 7003. The first submodule 7001 is used to obtain the total number of processes in the OpenMPI cluster. The second submodule 7002 is used to calculate the index value of each task based on the OpenMP multi-threading method, and perform remainder processing on the difference between the total number of processes and a first predetermined value to obtain multiple remainders, wherein the first predetermined value is a non-zero positive integer. The third submodule 7003 is mainly used to allocate each task to the process whose index value is the corresponding remainder. The specific operations performed by the task allocation module 700 and the submodules it contains can be referred to the above method for S200 and Figure 3 The relevant description will not be described in detail here.

[0138] The label assignment module 701 is mainly used to assign labels to the above-mentioned multiple tasks, wherein different tasks have different labels. In one example, the label assignment module 701 can assign first labels to the content sizes of multiple tasks based on the OpenMP multi-threading method, and assign second labels to the content of multiple tasks, wherein the first labels of any two tasks are different, the second labels of any two tasks are different, the first label and the second label of the same task are different, and the first label of any task and the second label of any task are different, wherein the first label is used to make the corresponding process of the receiver wait for the content size of the task based on the first label, and wherein the second label is used to make the corresponding process of the receiver wait for the content of the task based on the second label. In another example, for any task, the label assignment module 701 can calculate the first label of the task based on the index value of the task and the preset label starting value based on a thread of OpenMP, and use a label that is sequentially continuous with the first label as the second label of the task. The specific operations performed by the label assignment module 701 can refer to the relevant description of S201 in the above method, which will not be described in detail here.

[0139] The first label handshake module 702 is mainly used to send the corresponding relationship information of the task, process and label to the task receiver, so that the process in each task receiver waits to receive the corresponding task according to the label. Optionally, the first label handshake module 702 may include: a fourth submodule 7021, a fifth submodule 7022 and a sixth submodule 7023. The fourth submodule 7021 is used to form a one-dimensional array with the index value of each task, the index value of the process corresponding to each task, and the first label and the second label respectively assigned to each task. The fifth submodule 7022 is used to determine the index value of the label handshake process corresponding to each task according to the index value of each task. The sixth submodule 7023 is used to send the one-dimensional array corresponding to each task according to the index value of each label handshake process based on the OpenMP multi-threading method, so that the corresponding process in the task receiver waits to receive the content size and content of the task according to the first label and the second label in the corresponding relationship information obtained by the label handshake process; wherein, for any task, the index value of the label handshake process corresponding to the task is different from the index value of the process assigned to the task. The specific operations performed by the first tag handshake module 702 and its submodules can be referred to in the above method for S202 and Figure 4 The relevant description will not be described in detail here.

[0140] The first sending module 703 is mainly used to send the content size of the multiple tasks to each task receiver based on the OpenMP multi-threading method according to the correspondence information between the tasks and the labels, so that the corresponding processes in each task receiver respectively set the buffer area according to the received content size. For example, the first sending module 703 can send the content size of each task and the first label of each task to each task receiver based on the OpenMP multi-threading asynchronous sending method according to the correspondence information between the tasks and the first label. The specific operations performed by the first sending module 703 can refer to the relevant description of S203 in the above method, and will not be described in detail here.

[0141] The second sending module 704 is mainly used to send the contents of the multiple tasks to each task receiver based on the OpenMP multi-threading method according to the correspondence information between the task and the label, so that the corresponding process in each task receiver receives the task according to the label and stores it in the corresponding buffer area. For example, the second sending module 704 can send the contents of each task and the second label of each task to each task receiver based on the OpenMP multi-threaded asynchronous sending method according to the correspondence information between the task and the second label. The specific operations performed by the second sending module 704 can refer to the relevant description of S204 in the above method, which will not be described in detail here.

[0142] Figure 8 FIG. 1 is a schematic diagram of the structure of another embodiment of the multi-threaded data transceiver device disclosed in the present invention. The device of this embodiment can be used to implement the corresponding method embodiment of the present invention. Figure 8 The device shown includes: a second tag handshake module 800 , a waiting receiving module 801 , a buffer area setting module 802 , a task storage module 803 and a task execution module 804 .

[0143] The second tag handshake module 800 is mainly used to receive the correspondence information of the task, process and tag from the task sender. The specific operations performed by the second tag handshake module 800 can refer to the relevant description of S500 in the above method, and will not be described in detail here.

[0144] The waiting receiving module 801 is mainly used to control the corresponding process based on OpenMPI to be in a waiting receiving state on the corresponding label according to the above correspondence information. The specific operations performed by the waiting receiving module 801 can refer to the relevant description of S501 in the above method, which will not be described in detail here.

[0145] The buffer area setting module 802 is mainly used for, when the process receives the content size of the task from the task sender based on the OpenMP multithreading method according to the label in the above correspondence information, to enable the process to perform the operation of setting the buffer area according to the content size of the task. The specific operation performed by the buffer area setting module 802 can refer to the relevant description of S502 in the above method, which will not be described in detail here.

[0146] The task storage module 803 is mainly used for, when the process receives the content of the task from the task sender based on the OpenMP multithreading method according to the label in the corresponding relationship information, the process stores the content of the task in the cache area set by the cache area setting module 802. The specific operations performed by the task storage module 803 can be found in the relevant description of S503 in the above method, which will not be described in detail here.

[0147] The task execution module 804 is mainly used to make the process execute the task according to the content of the task stored in the buffer area set by the buffer area setting module 802, and make the process return the execution result of the task to the task sender according to the label in the corresponding relationship information. The specific operation performed by the task execution module 804 can refer to the relevant description of S504 in the above method, and will not be described in detail here.

[0148] Exemplary Electronic Devices

[0149] Reference below Fig. 9 An electronic device according to an embodiment of the present disclosure is described. Fig. 9 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Fig. 9 As shown, the electronic device 91 includes one or more processors 911 and a memory 912 .

[0150] The processor 911 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 91 to perform desired functions.

[0151] The memory 912 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, may include: random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory, for example, may include: read-only memory (ROM), hard disk, and flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 911 may execute the program instructions to implement the multi-threaded data transceiving method based on the various embodiments of the present disclosure described above and / or other desired functions.

[0152] In one example, the electronic device 91 may also include: an input device 913 and an output device 914, etc., which are interconnected through a bus system and / or other forms of connection mechanisms (not shown). In addition, the input device 913 may also include, for example, a keyboard, a mouse, etc. The output device 914 may output various information to the outside. The output device 914 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0153] Of course, to simplify, Fig. 9 Only some of the components related to the present disclosure in the electronic device 91 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 91 may also include any other appropriate components.

[0154] Exemplary computer program products and computer-readable storage media

[0155] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the multi-threaded data sending and receiving method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0156] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0157] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the multi-threaded data sending and receiving method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0158] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive enumeration) of readable storage media can include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0159] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0160] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0161] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, and configured in any manner. Words such as "include," "comprise," "have," and the like are open words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0162] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0163] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0164] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, etc., will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0165] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A multi-threaded data sending and receiving method, which is executed on a task sender, comprising: Distribute multiple tasks to multiple processes based on the OpenMPI open message transmission interface; Assigning labels to the multiple tasks respectively, wherein different tasks have different labels; Sending the corresponding relationship information of the task, process and label to the task receiver, so that the process in each task receiver waits to receive the corresponding task according to the label; According to the correspondence information between the tasks and the labels, based on the multi-processing and multi-threading method open by OpenMP, the content sizes of the multiple tasks are respectively sent to each task receiver, so that the corresponding process in each task receiver respectively sets the buffer area according to the received content size; According to the correspondence information between the tasks and the labels, based on the OpenMP multithreading method, the contents of the multiple tasks are sent to each task receiver respectively, so that the corresponding process in each task receiver respectively receives the tasks according to the labels and stores them in the corresponding buffer area.

2. The method according to claim 1, wherein: The method of allocating multiple tasks to multiple processes based on OpenMPI includes: Distribute multiple tasks to multiple different processes in an OpenMPI cluster; Among them, the processes assigned to any two tasks are not the same.

3. The method according to claim 2, wherein: The method of allocating the multiple tasks to multiple different processes in the OpenMPI cluster includes: Get the total number of processes in the OpenMPI cluster; Based on the OpenMP multithreading method, the index value of each task is calculated, and the difference between the total number of processes and the first predetermined value is respectively subjected to modulo processing to obtain a plurality of remainders, wherein the first predetermined value is a non-zero positive integer; Assign each task to the process with the corresponding remainder as index value.

4. The method according to claim 1, wherein: The assigning labels to the plurality of tasks respectively includes: Based on the OpenMP multithreading method, first tags are respectively assigned to the content sizes of the plurality of tasks, and second tags are respectively assigned to the contents of the plurality of tasks; Among them, the first tags of any two tasks are different, the second tags of any two tasks are different, the first tag and the second tag of the same task are different, the first tag of any task and the second tag of any task are different, the first tag is used to make the corresponding process of the recipient wait for the content size of the received task based on the first tag, and the second tag is used to make the corresponding process of the recipient wait for the content of the received task based on the second tag.

5. The method according to claim 4, wherein: The step of assigning first tags to the content sizes of the plurality of tasks respectively, and assigning second tags to the contents of the plurality of tasks respectively, comprises: For any task, a thread based on OpenMP calculates the first label of the task according to the index value of the task and the preset label start value, and uses a label sequentially following the first label as the second label of the task.

6. The method according to claim 4 or 5, wherein: The sending the corresponding relationship information of the task, process and label to the task receiver includes: The index value of each task, the index value of the process corresponding to each task, and the first label and the second label respectively assigned to each task are respectively formed into a one-dimensional array; According to the respective index values ​​of the tasks, the index values ​​of the label handshake processes corresponding to the respective tasks are determined; Based on the OpenMP multi-threading method, according to the index value of each label handshake process, the one-dimensional array corresponding to each task is sent, so that the corresponding process in the task receiver waits to receive the content size and content of the task according to the first label and the second label in the corresponding relationship information obtained by the label handshake process; For any task, the index value of the tag handshake process corresponding to the task is different from the index value of the process allocated to the task.

7. The method according to claim 4 or 5, wherein: According to the correspondence information between the tasks and the labels, based on the OpenMP multithreading method, the content size of the multiple tasks is sent to each task receiver respectively, including: According to the correspondence information between the task and the first label, based on the OpenMP multi-threaded asynchronous sending method, the content size of each task and the first label of each task are sent to each task receiver respectively; According to the correspondence information between the tasks and the labels, based on the OpenMP multithreading method, the contents of the multiple tasks are respectively sent to each task receiver, including: According to the correspondence information between the task and the second tag, based on the asynchronous sending mode of OpenMP multi-threading, the content of each task and the second tag of each task are sent to each task receiver respectively.

8. A multi-threaded data sending and receiving method, the method being executed on a task receiving side, comprising: Receive the corresponding relationship information of tasks, processes and tags from the task sender; According to the corresponding relationship information, the corresponding process based on OpenMPI is placed in a waiting-for-receiving state on the corresponding label; When the process receives the content size of the task from the task sender based on the label in the corresponding relationship information and based on the OpenMP multi-threading method, the process sets the buffer area according to the content size of the task; When the process receives the content of the task from the task sender based on the label in the corresponding relationship information and based on the OpenMP multi-threading method, the process stores the content of the task in the buffer area; The process executes the task according to the content of the task stored in the buffer area, and returns the execution result of the task to the task sender according to the tag in the corresponding relationship information.

9. A multi-threaded data transceiver device, comprising: Task allocation module, used to allocate multiple tasks to multiple processes based on OpenMPI; A label assignment module, used to assign labels to the plurality of tasks respectively, wherein different tasks have different labels; A first label handshake module is used to send the corresponding relationship information of the task, process and label to the task receiver, so that the process in each task receiver waits to receive the corresponding task according to the label; A first sending module is used to send the content size of the multiple tasks to each task receiver respectively according to the correspondence information between the tasks and the labels based on the OpenMP multi-threading method, so that the corresponding process in each task receiver respectively sets the buffer area according to the received content size; The second sending module is used to send the contents of the multiple tasks to each task receiver based on the OpenMP multi-threading method according to the corresponding relationship information between the tasks and the labels, so that the corresponding process in each task receiver receives the tasks according to the labels and stores them in the corresponding cache area.

10. The device according to claim 9, wherein: The task allocation module is specifically used for: Distribute multiple tasks to multiple different processes in an OpenMPI cluster; Among them, the processes assigned to any two tasks are not the same.

11. The device according to claim 10, wherein: The task allocation module includes: The first submodule is used to obtain the total number of processes in the OpenMPI cluster; The second submodule is used to calculate the index value of each task based on the OpenMP multi-threading method, and perform remainder processing on the difference between the total number of processes and the first predetermined value to obtain multiple remainders, wherein the first predetermined value is a non-zero positive integer; The third submodule is used to assign each task to the process whose index value is the corresponding remainder.

12. The device according to claim 9, wherein: The label allocation module is specifically used for: Based on the OpenMP multithreading method, first tags are respectively assigned to the content sizes of the plurality of tasks, and second tags are respectively assigned to the contents of the plurality of tasks; Among them, the first tags of any two tasks are different, the second tags of any two tasks are different, the first tag and the second tag of the same task are different, the first tag of any task and the second tag of any task are different, the first tag is used to make the corresponding process of the recipient wait for the content size of the received task based on the first tag, and the second tag is used to make the corresponding process of the recipient wait for the content of the received task based on the second tag.

13. The device according to claim 12, wherein: The label allocation module is specifically used for: For any task, a thread based on OpenMP calculates the first label of the task according to the index value of the task and the preset label start value, and uses a label sequentially following the first label as the second label of the task.

14. The device according to claim 12 or 13, wherein: The first tag handshake module includes: The fourth submodule is used to form a one-dimensional array with the index value of each task, the index value of the process corresponding to each task, and the first label and the second label respectively allocated to each task; The fifth submodule is used to determine the index value of the label handshake process corresponding to each task according to the index value of each task; The sixth submodule is used to send the one-dimensional array corresponding to each task according to the index value of each label handshake process based on the OpenMP multi-threading method, so that the corresponding process in the task receiver waits to receive the content size and content of the task according to the first label and the second label in the corresponding relationship information obtained by the label handshake process; For any task, the index value of the tag handshake process corresponding to the task is different from the index value of the process allocated to the task.

15. The device according to claim 12 or 13, wherein: The first sending module is specifically used for: According to the correspondence information between the task and the first label, based on the OpenMP multi-threaded asynchronous sending method, the content size of each task and the first label of each task are sent to each task receiver respectively; The second sending module is specifically used for: According to the correspondence information between the task and the second tag, based on the asynchronous sending mode of OpenMP multi-threading, the content of each task and the second tag of each task are sent to each task receiver respectively.

16. A multi-threaded data transceiver device, comprising: The second label handshake module is used to receive the corresponding relationship information of the task, process and label from the task sender; A waiting receiving module, used for making the corresponding process based on OpenMPI be in a waiting receiving state on the corresponding label according to the corresponding relationship information; A buffer area setting module is used to enable the process to set a buffer area according to the content size of the task when the process receives the content size of the task from the task sender based on the OpenMP multi-threading method according to the label in the corresponding relationship information; A task storage module, configured to enable the process to store the content of the task in the cache area when the process receives the content of the task from the task sender based on the label in the corresponding relationship information in an OpenMP multi-threaded manner; The task execution module is used to enable the process to execute the task according to the content of the task stored in the cache area, and to enable the process to return the execution result of the task to the task sender according to the label in the corresponding relationship information.

17. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 8.

18. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 8.

Images