A data exchange method based on dynamic process calculation

Abstract

The present invention discloses a data exchange method based on dynamic process calculation. In view of the fact that existing data exchange methods in dynamic process calculation are difficult to achieve fast and accurate data exchange and process synchronization between heterogeneous programs, the present invention proposes a data exchange and process control method. First, a data exchange method is designed based on an in-memory database to achieve data exchange between different application processes in a dynamic process. Second, considering the process synchronization problem of multiple programs, a process control mechanism is designed. When the program process requests empty data, the process can be controlled to enter a waiting state, and the request result is returned after the data request is successful. Finally, the present invention solves the problem of data exchange and data synchronization between different programs in dynamic process calculation, which has higher practicality and application value.

Description

Technical Field

[0001] The present invention belongs to the field of high performance computing, and in particular relates to a data exchange method for dynamic process computing in high performance computing. Background Art

[0002] In dynamic process computing, in order to improve computing efficiency, the current common method is block parallel computing, that is, splitting a dynamic computing process into multiple subroutines and deploying these subroutines in a distributed system. Multiple subroutines can be parallelized in the same time slot.

[0003] When executing coupled program computations, it's important to consider data exchange between different program processes. Commonly used data exchange methods in parallel computing include the Message Passing Interface (MPI) and shared memory. MPI data exchange is an inter-process data exchange method. While it can address data exchange between different processes in dynamic computations, it struggles with data exchange between different programs or processes on different compute nodes. Shared memory data exchange allows data sharing between different processes without requiring data transfer, significantly improving data transfer efficiency. However, shared memory data exchange lacks a synchronization mechanism, and inter-process communication using shared memory often requires other means of synchronization.

[0004] Based on this, the present invention discloses a data exchange method based on dynamic process calculation, which mainly solves the problem of data exchange and process synchronization between different programs on multiple computing nodes. Summary of the Invention

[0005] To address the difficulty of achieving fast and accurate data exchange and process synchronization between heterogeneous programs using existing data exchange methods in dynamic process computing, this paper proposes a data exchange method and a process control method. Firstly, a data exchange method based on an in-memory database is designed to enable data exchange between different application processes within a dynamic process. Secondly, considering the process synchronization issues of multiple programs, a process control mechanism is designed. When a program process requests empty data, it can control the process to enter a waiting state and return the request result after the data request is successful.

[0006] The data exchange and process control method proposed by the present invention solves the problem of data exchange and data synchronization between different programs in dynamic process calculation, and has strong practicality and high application value.

[0007] In a first aspect, the present invention provides a data exchange method based on dynamic process calculation, characterized in that the method comprises the following steps:

[0008] Step 1: Input initial parameters;

[0009] Step 2: Calculate;

[0010] Step 3: Write data;

[0011] Step 4: Data exchange;

[0012] In step 3, a unique identifier for the exchanged data is generated. The key structure of the identifier includes: a fixed identifier, SEND_PID, SEND_MYID, REC_PID, REC_MYID, and CURRENT. Among them, the fixed identifier consists of 5 - 10 letters or numbers. SEND_PID is the program number for sending data, SEND_MYID is the process number for sending data, REC_PID is the program number for receiving data, REC_MYID is the process number for receiving data, and CURRENT indicates the position of the current exchanged data in the length KLEN.

[0013] As a further improved technical solution, step 3 specifically includes:

[0014] Step 31: Obtain the data length KLEN data identification information;

[0015] Step 32: Invoke the database connection interface;

[0016] Step 33: Connect to the in - memory database;

[0017] Step 34: Generate a unique identifier for the exchanged data;

[0018] Step 35: According to the unique identifier of the exchanged data, write the data into the in - memory database. If the written data j < KLEN, execute step 34, and loop to perform the data writing operation until all the required data is written.

[0019] As a further improved technical solution, step 4 specifically includes:

[0020] Step 41: Obtain the data length KLEN data identification information.

[0021] Step 42: Invoke the database connection interface.

[0022] Step 43: Connect to the in - memory database.

[0023] Step 44: Generate a unique identifier for the exchanged data.

[0024] Step 45: Obtain the swapped data of the specified program process by exchanging the data unique identifier, and determine whether the data has been stored in the memory database. If the data query is successful, that is, read the data in the memory database and perform the data reading of the memory database. If the read data j < KLEN, execute Step 44, and loop to perform the data reading operation until all the required data is read.

[0025] As a further improved technical solution, Step 45 further includes: if the data query is successful, immediately return the data to the program; otherwise, use the process control mechanism for process synchronization.

[0026] As a further improved technical solution, the process control mechanism includes:

[0027] S1: Process 1 of Program A on computing node 1 issues a data exchange request.

[0028] S2: Initialize a timer to record the request time.

[0029] S3: Query the required data from the memory database according to the data exchange request.

[0030] S4: When the result is queried, directly return the result to Process 1 of Program A on computing node 1; otherwise, execute S5.

[0031] S5: When the query result is empty, repeat S3 every Δt to query data from the database, and at the same time, determine whether the data t recorded by the timer exceeds the maximum waiting time T allowed by the program. If it does not exceed, execute S2; if it exceeds, return a null value to Process 1 of Program A on computing node 1.

[0032] As a further improved technical solution, while Process 1 of Program A on computing node 1 requests to exchange data, other processes can request to read or write data to the memory database at any time.

[0033] As a further improved technical solution, the current swapped data is integer or real.

[0034] As a further improved technical solution, the current swapped data is stored in an array with a length of KLEN, and CURRENT represents the position of the current swapped data in the array.

[0035] In a second aspect, the present invention provides a computer device, including a memory and a processor. The memory stores a computer program, and it is characterized in that when the processor executes the computer program, it implements the data exchange method based on dynamic process calculation described above.

[0036] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned data exchange method based on dynamic process calculation.

[0037] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0038] 1. The present invention designs a data exchange method based on an in-memory database to realize data exchange between different application processes in a dynamic process. The method is simple in design and has high data exchange efficiency.

[0039] 2. Considering the process synchronization problem of multiple programs, this invention designs a process control mechanism. When a program process requests empty data, it can control the process to enter a waiting state and return the request result after the data request is successful. The method provided by this invention solves the problem of data exchange and synchronization between different programs in dynamic process computing, and has strong practicality and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a diagram of the data exchange mechanism architecture provided by the present invention;

[0041] Figure 2 The program data exchange coupling model provided by the present invention;

[0042] Figure 3 The data exchange flow chart provided by the present invention;

[0043] Figure 4 The data writing flow chart provided by the present invention;

[0044] Figure 5 The data reading flow chart provided by the present invention;

[0045] Figure 6 This is a diagram of the process control mechanism provided by the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the embodiments shown in the accompanying drawings.

[0047] Example 1

[0048] In order to solve the problem of data exchange of calculation results between multiple program processes in dynamic process calculation, the present invention proposes a data exchange mechanism based on memory database, such as Figure 1 As shown, Figure 1This is the architecture diagram of the data exchange mechanism provided by the present invention. The data exchange mechanism comprises a database server 1, a switch 3, N application computing servers, and multiple data transmission links. An in-memory database 2 is deployed on database server 1, and switch 3 forwards data between the in-memory database 2 and the application computing servers, as well as between the application computing servers, to reduce the impact of transmission latency on computing efficiency.

[0049] The data exchange mechanism architecture further includes multiple data transmission links, wherein 4, 6, 8 ... M-1 are data transmission downlinks respectively; 5, 7, 9 ... M are data transmission uplinks respectively.

[0050] M+1, M+2...M+N are dynamic process computing servers, on which one or more applications with different functions are deployed respectively. Each application participates in part of the calculation in the dynamic computing process. Different servers are connected through a network, such as through a 20G Ethernet cable and a switch, to realize data communication between different applications.

[0051] To illustrate the dynamic process of data exchange Dy, we take the programs (A, B, C, D) running on different servers as an example, Dy = {A, B, C, D}, where A, B, C, D are different programs running on different servers, and Dy represents the set of data that needs to be exchanged between them. Figure 2 As shown, the exchange data between program A and program B running on different servers is composed of the set Indicates that the elements in the exchange data set are the data that need to be exchanged between different programs, and LA represents the length of the data that needs to be exchanged between program A and program B. The exchange data between program B and program C running on different servers is represented by the set LB represents the length of data that needs to be exchanged between program B and program C. The exchanged data between program C and program D running on different servers is represented by the set Indicates that LC represents the length of data that needs to be exchanged between program C and program D.

[0052] It should be noted that different programs running on different servers run independently when no data exchange request is issued. When a data exchange request is issued, the program is in a waiting state until all data that meets the request is received and then continues to enter the computing state.

[0053] Taking the data exchange between program A and program B as an example, the data exchange process provided by the present invention is as follows: Figure 3 The specific steps are as follows:

[0054] Step 1: Input of initial parameters.

[0055] Programs A and B input or initialize the initial values ​​of relevant parameters according to their respective requirements.

[0056] Step 2: Calculation.

[0057] Programs A and B perform independent computations. These computations can be performed in parallel, or each program can be divided into multiple processes to perform independent computations.

[0058] Step 3: Data writing.

[0059] If the length of the data to be written (or the number of data to be written) is K, i is used as the loop condition, and each time data is written, i=i+1; if i>K, the data writing process is terminated.

[0060] It should be noted that different programs or processes need to exchange data with each other during operation. Since different programs are assigned different amounts of data to calculate, the completion time of a single calculation is also different. Assuming that program B completes the calculation first, after program B completes the calculation, it needs to exchange data Q. AB Write to the in-memory database.

[0061] During the data writing process, since commonly used in-memory databases access data based on key values, a unique identifier "key" for data exchange needs to be generated here.

[0062] We designed a unique identifier KEY AB , used to identify data exchanged between programs. The identification element Depend on<FI_EXG_D,KLEN,SEND_PID,SEND_MYID,REC_PID,REC_MYID> FI_EXG_D represents the first address of the exchange data array, KLEN is the exchange data length or the number of exchange data, SEND_PID is the program number for sending data, SEND_MYID is the process number for sending data, REC_PID is the program number for receiving data, and REC_MYID is the process number for receiving data.

[0063] However, for the above unique identification, since different programs or processes may run on different computing nodes, the first address of their exchange data array changes due to the change of storage node or storage location, which can easily cause errors when accessing data, resulting in access data identification.

[0064] On this basis, the present invention proposes a new exchange data unique identifier for identifying the current exchange data, the structure of which is as follows:

[0065] Fixed logo SEND_PID SEND_MYID REC_PID REC_MYID CURRENT

[0066] Among them, the fixed identifier consists of 5 - 10 letters or data and can be set manually. SEND_PID, SEND_MYID, REC_PID, REC_MYID, and CURRENT consist of 1 - 4 decimal data words. SEND_PID is the program number of the sent data, SEND_MYID is the process number of the sent data, REC_PID is the program number of the received data, REC_MYID is the process number of the received data, and CURRENT indicates the position of the currently exchanged data. The unique identifier for exchanging data provided by the present invention identifies the position of the currently exchanged data through CURRENT, that is, a value assigned by the system is used as an indication of the position of the currently exchanged data, facilitating the system to access and store this data.

[0067] Preferably, the currently exchanged data is integer or real type.

[0068] Preferably, the currently exchanged data can be stored in an array with a length of KLEN (that is, KLEN data can be stored), and CURRENT identifies the index of the currently exchanged data, indicating the position or index of the currently exchanged data in the array with a length of KLEN.

[0069] In this embodiment, the data writing process in step 3 is as Figure 4 shown. This process is mainly divided into three steps: connecting to the in - memory database, generating a unique identifier for exchanging data, and data persistence. Specifically, it includes:

[0070] Step 31: Obtain the data length KLEN and data identification information.

[0071] Preferably, KLEN is the number of data to be obtained.

[0072] Step 32: Invoke the database connection interface.

[0073] Step 33: Connect to the in - memory database.

[0074] Step 34: Generate a unique identifier for exchanging data with a unique identifier in multiple programs.

[0075] Generate a "key" with a unique identifier in multiple programs - the unique identifier structure for exchanging data.

[0076] Step 35: According to the unique identifier for exchanging data, the specific data in the program can be written into the in - memory database. If the written data count j < KLEN, execute step 34, and loop to perform the data writing operation until all the required data is written. Here, j is used for counting the written data, and after each write operation is completed, j = j + 1.

[0077] Step 4: Data exchange.

[0078] For the specific steps of data exchange, see Figure 5 , Figure 5 The data exchange flow chart is shown as follows. According to Figure 5 shown, step 4 specifically includes:

[0079] Step 41: Obtain the data length KLEN and data identification information.

[0080] Preferably, KLEN is the quantity of data to be obtained.

[0081] Step 42: Invoke the database connection interface.

[0082] Step 43: Connect to the in-memory database.

[0083] Step 44: Generate a unique identifier for the exchanged data.

[0084] Step 45: Obtain the exchanged data of the specified program process through the unique identifier of the exchanged data, and determine whether the data has been stored in the in-memory database. If the data query is successful, that is, read the data in the in-memory database and perform the data reading operation of the in-memory database. If the read data j < KLEN, execute step 44, and loop to execute the data reading operation until all the required data is read completely.

[0085] Among them, j is used for counting the read data. After each read operation is completed, j = j + 1.

[0086] Embodiment 2

[0087] Based on Embodiment 1, different from the data reading methods of other application programs, when the data query fails, it cannot return an empty result set as the program response. Instead, process control is required, and the next operation can only be performed after waiting for other programs to write the requested data, so as to achieve synchronization between processes.

[0088] Considering the time difference in the calculation results of different programs, when obtaining the exchanged data in step 45 of Embodiment 1, the obtained result may be empty. To solve the synchronization problem of data exchange between different program processes, the present invention designs a process control mechanism to achieve the process synchronization problem in process data exchange.

[0089] Specifically, the program that executes the data request connects to the in-memory database, and searches for data in the in-memory database according to the process's request for data. As Figure 5 shown, if the data is obtained, the data is immediately returned to the program. Otherwise, a process control mechanism is used for process synchronization.

[0090] The process of process synchronization is as Figure 6 shown, and specifically includes:

[0091] Step 1: Process 1 of program A on computing node 1 issues a data exchange request.

[0092] Step 2: Initialize a timer to record the request time.

[0093] Step 3: Based on the data exchange request, query the required data from the in-memory database.

[0094] Step 4: When the query result is found, the result is directly returned to process 1 of program A on computing node 1. Otherwise, execute step 5.

[0095] Step 5: When the query result is empty, repeat step 3 every Δt to query data from the database and determine whether the timer record data t exceeds the maximum waiting time T allowed by the program. If not, execute step 3. If it exceeds, return a null value to process 1 of program A on computing node 1.

[0096] It should be noted that while process 1 of program A on computing node 1 requests to exchange data, other processes, such as process 3 of program B on computing node 2, can write the corresponding value to the specified "key" in the in-memory database at any time. Moreover, any data request of any program process will be responded to by other program processes (with different response times). Therefore, data exchange will not fall into a "deadlock" state.

[0097] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0098] The scope of protection of the present invention is not limited to the above-described embodiments. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the scope and spirit of the present invention. If such modifications and variations fall within the scope of the claims of the present invention and their equivalents, the present invention is intended to include such modifications and variations.

Claims

1. A data exchange method based on dynamic process calculation, characterized in that: The method includes the following steps: Step 1: Input initial parameters; Step 2: Calculate; Step 3: Write data; Step 4: Data exchange; In Step 3, a unique identifier for the exchanged data is generated. The key structure of the identifier includes: a fixed identifier, SEND_PID, SEND_MYID, REC_PID, REC_MYID, and CURRENT. Among them, SEND_PID is the program number of the sending data, SEND_MYID is the process number of the sending data, REC_PID is the program number of the receiving data, REC_MYID is the process number of the receiving data, and CURRENT represents the position of the current exchanged data; Step 3 specifically includes: Step 31: Obtain the data length KLEN and data identification information; Step 32: Call the database connection interface; Step 33: Connect to the in-memory database; Step 34: Generate a unique identifier for the exchanged data; Step 35: According to the unique identifier of the exchanged data, write the data into the in-memory database. If the written data length j < KLEN, execute Step 34, and loop to execute the data writing operation until all the required data is written; Step 4 specifically includes: Step 41: Obtain the data length KLEN and data identification information; Step 42: Call the database connection interface; Step 43: Connect to the in-memory database; Step 44: Generate a unique identifier for the exchanged data; Step 45: Obtain the exchanged data of the specified program process through the unique identifier of the exchanged data, and determine whether the data has been stored in the in-memory database. If the data query is successful, that is, read the data in the in-memory database, and execute the data reading in the in-memory database. If the read data length j < KLEN, execute Step 44, and loop to execute the data reading operation until all the required data is read.

2. The method according to claim 1, wherein: Step 45 further includes: If the data query is successful, immediately return the data to the program, otherwise, use the process control mechanism for process synchronization.

3. The method according to claim 2, wherein: The process control mechanism includes: S1: Process 1 of Program A on Computing Node 1 issues a data exchange request; S2: Initialize a timer to record the request time; S3: Query the required data from the in-memory database according to the data exchange request; S4: When the result is queried, directly return the result to Process 1 of Program A on Computing Node 1, otherwise, execute S5; S5: When the query result is empty, , repeat S3, query data from the database, and at the same time determine whether the timer recorded data t exceeds the maximum waiting time T allowed by the program. If it exceeds, return a null value to process 1 of program A on computing node 1.

4. The method according to claim 3, wherein: While Process 1 of Program A on Computing Node 1 requests data exchange, other processes can request to read or write data from the in-memory database at any time.

5. The method according to claim 1, wherein: The current exchanged data is integer or real.

6. The method according to claim 1, wherein: The current exchanged data is stored in an array with a length of KLEN, and CURRENT represents the position of the current exchanged data in the array.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements the data exchange method based on dynamic process calculation described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the data exchange method based on dynamic process calculation described in any one of claims 1 to 6.

Images