Data processing method based on template stencil calculation and related equipment

Abstract

The invention provides a data processing method based on template stencil calculation and related equipment, and relates to the field of computers. The method is applied to a cluster computing system, the cluster computing system comprises a plurality of processes, and the processes are used for processing data of a cluster application. Wherein each process is used for processing a plurality of data, a section of memory address space is distributed to each process, and the method comprises the following steps of: uniformly addressing the address spaces of the plurality of processes to form a uniform address space, and recording a corresponding relationship between the uniform address space and the memory address spaces of the plurality of processes. And responding to a command of acquiring the first data of the second process by the first process in the plurality of processes, and acquiring a first address of the first data in the uniform address space according to the information of the first data. And obtaining a second address in a memory address space of a second process corresponding to the first address according to the first address. And reading the first data from the memory address space of the second process according to the second address.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data processing method and related equipment based on stencil calculation. Background Art

[0002] Stencil calculation is a cyclic operation mode, which is characterized by traversing the entire data area and performing the same calculation operation on each data in the data area. As the most common calculation mode in many application fields, stencil calculation often occupies a core hotspot position. Its calculation process requires obtaining the data of neighboring grid points according to the numerical format or template to update the data, so it involves a large amount of node communication and data transmission.

[0003] Specifically, a process processes multiple data in a data area, and the multiple data form grid data. The grid data includes internal grid points and edge grid points in the top row, the bottom row, the leftmost row, or the rightmost row of the grid. During the data update and iteration process, each grid point needs to perform stencil calculation based on the initial value of the neighboring grid points to obtain the updated value of each grid point. The neighboring grid points of the internal grid points are all within the grid, and their stencil calculation only requires the data within this process. The edge grid points may have neighboring grid points belonging to other processes, and must communicate with other processes to obtain data on other processes.

[0004] Currently, each process relies on calling the multi-point interface (MPI) for communication. Since the process needs to switch from user mode to kernel mode during data communication, when updating grid data, the process needs to wait for all communications with other processes to be completed before performing stencil calculations on each edge grid point. This will cause delays in updating edge grid point data, seriously affecting the efficiency of stencil calculations. Summary of the invention

[0005] The embodiment of the present application provides a data processing method and related equipment based on stencil calculation. In the embodiment of the present application, the address space corresponding to each process is uniformly addressed to form a unified address space, and the memory address space of multiple processes is mapped to the unified address space. In this way, each process shares the unified address space and can directly access the memory address space corresponding to other processes to read data on other processes. Therefore, the data update delay caused by MPI communication can be avoided, and the computational efficiency of stencil calculation can be improved.

[0006] In the first aspect, an embodiment of the present application provides a data processing method based on stencil calculation, which is applied in a cluster computing system. Specifically, multiple processes are running on the cluster computing system to jointly process the data of the cluster application. The kernel allocates a memory address space to each process to store the data in the process. When performing stencil calculation, the address spaces of all processes are first uniformly addressed to form a virtual unified address space. Then the memory address space corresponding to each process is mapped to the unified address space, and the correspondence between the memory address space corresponding to each process and the unified address space is recorded. Then when the first process needs to obtain the first data on the second process, the space address of the first process in the formed unified address space is first determined, and then based on the correspondence between the memory address space corresponding to each process and the unified address space, the second address in the memory address space of the second process corresponding to the space address is determined. Then the memory address space of the second process is directly accessed according to the second address to read the first data.

[0007] In an embodiment of the present application, multiple processes share a unified address space formed based on unified addressing, and can directly obtain the memory address of a certain data in the memory address space of other processes based on the correspondence between the unified address space and the memory address space of each process. In this way, a process can directly access the memory address space of other processes with the memory address corresponding to the data, and can directly obtain the data without relying on MPI communication. In this way, data update delays caused by MPI communication can be avoided, and the computational efficiency of stencil calculations can be improved.

[0008] In an optional implementation, the data of the cluster application processed by multiple processes are arranged in a certain order, and the size of each data is the same. Then, when the address spaces of multiple processes are uniformly addressed, the first address of the first data in the data of the cluster application can be shared with multiple processes. In this way, when the memory address space of each process is addressed, a new virtual address of the memory address space of each process needs to be formed based on the first address. It can be understood that the first address is the first address of the unified address space formed.

[0009] In an optional embodiment, when a process needs to obtain the first data on other processes, it is necessary to first determine the corresponding address offset based on the order of the first address in the data of the cluster application and the data of each cluster application, and then determine the first address of the first data in the data of the cluster application and the address offset.

[0010] In an optional implementation, after determining the first address of the first data in the unified address space, it is necessary to determine the second address of the memory address space of a process corresponding to the first address according to the correspondence between the unified address space and the memory address space of each process. It can be understood that the virtual address in the unified address space corresponds one-to-one to the memory address space of each process.

[0011] In an optional implementation, the data to be processed by each process can be mapped to a grid to form grid data, where each data corresponds to a grid point. Then, when the processes communicate with each other, they generally need to obtain the data of the top row, the bottom row, the leftmost row, or the rightmost row of grid points in the network data corresponding to other processes.

[0012] In a second aspect, an embodiment of the present application provides a data processing device based on stencil calculation, which is applied to a cluster computing system. The cluster computing system includes multiple processes, and the multiple processes are used to process data of cluster applications. Each process processes multiple data, and each process is allocated a memory address space. The data processing device includes:

[0013] The processing unit is used to uniformly address the address spaces of multiple processes to form a unified address space, and record the corresponding relationship between the unified address space and the memory address spaces of multiple processes.

[0014] The acquisition unit is used to respond to a command from a first process among multiple processes to acquire first data of a second process, and acquire a first address of the first data in a unified address space according to information of the first data.

[0015] The acquiring unit is further configured to acquire, according to the first address, a second address in the memory address space of the second process corresponding to the first address.

[0016] The reading unit is used to read the first data from the memory address space of the second process according to the second address.

[0017] In an optional implementation, the data of the cluster application is data arranged in a certain order, and each data has the same size. The data updating device further includes a sending unit.

[0018] The sending unit is used to send the first address of the first data in the data of the cluster application to multiple processes, so that each of the multiple processes uses the first address as the first address of the unified address.

[0019] In an optional implementation, the information of the first data is the order of the first data in the data of the cluster application.

[0020] The acquisition unit is specifically used to determine the first address according to the order of the first data in the data of the cluster application, the first address of the unified address space and the size of each data.

[0021] In an optional implementation, the acquisition unit is specifically configured to determine the second address according to the first address and a correspondence between the unified address space and the memory address spaces of the multiple processes.

[0022] In an optional embodiment, the plurality of data processed by the first process forms grid data, wherein each data in the plurality of data processed by the first process corresponds to a grid point in the grid data, and the first data is data in the grid data at the top row, the bottom row, the leftmost row, or the rightmost row of grid points.

[0023] In a third aspect, an embodiment of the present application provides a processing device, including: a processor and a memory. The processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor to enable the processing device to implement any one of the data processing methods based on stencil calculation provided in the first aspect.

[0024] In a fourth aspect, an embodiment of the present application provides a chip, comprising: a processor and an interface circuit; the interface circuit is used to receive code instructions and transmit them to the processor; the processor is used to run the code instructions to execute any one of the data processing methods based on stencil calculation provided in the first aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer program instruction, and the computer program instruction is loaded and executed by a processor to implement any one of the data processing methods based on stencil calculation provided in the first aspect above.

[0026] In a sixth aspect, an embodiment of the present application provides a computer program product, including computer execution instructions. When the computer execution instructions are run on a computer, the computer executes any one of the data processing methods based on stencil calculation provided in the first aspect.

[0027] The technical effects brought about by any implementation method in the third aspect to the sixth aspect can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here.

[0028] In an embodiment of the present application, when multiple processes jointly process the data of a cluster application, the address spaces of the multiple processes are first uniformly addressed to form a virtual unified address space, and then the mapping relationship between each memory address space and the unified address space is recorded. Next, when multiple processes communicate, first respond to an instruction for a process to obtain data on another process, determine the first address of the required data corresponding to the instruction in the unified address space, then query the mapping relationship between each memory address space and the unified address space, determine the second address of the required data in the memory address space of another process based on the first address of the required data in the unified address space, and finally query the memory address space of another process according to the direct second address to read the required data. Since the memory address space corresponding to each process is uniformly addressed and the unified address space is shared, when the processes communicate to exchange data, they can directly access the memory address space of other processes according to the address of the data in the unified address space, without calling MPI communication to obtain data. In this way, when a process performs data calculation based on data on other processes, it does not need to wait for all communications to be completed before starting the calculation, but can achieve the overlap of communication and calculation, that is, calculation while communicating. Therefore, the embodiment of the present application can avoid the processing delay caused by MPI communication and improve the computing efficiency of data calculation. At the same time, it can also avoid multiple switching between the user state and the kernel state of the process caused by calling the MPI function for communication, thereby saving system resources and reducing the impact on the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the structure of the grid data provided in the embodiment of the present application;

[0030] Figure 2 A schematic diagram of another structure of grid data provided in an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of the process of stencil calculation in the prior art;

[0032] Figure 4 A system architecture diagram of a cluster computing system provided in an embodiment of the present application;

[0033] Figure 5 A system architecture diagram of an operating system provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of a flow chart of a data processing method based on stencil calculation provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of a flow chart of a data updating method based on stencil calculation provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of another structure of grid data provided in an embodiment of the present application;

[0037] Fig. 9 A flowchart of another data updating method based on stencil calculation provided in an embodiment of the present application;

[0038] Fig.10 A flowchart of another data updating method based on stencil calculation provided in an embodiment of the present application;

[0039] Fig.11 A schematic diagram of the structure of a data updating device based on stencil calculation provided in an embodiment of the present application;

[0040] Fig.12 A schematic diagram of the structure of a processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The embodiment of the present application provides a data processing method and related equipment based on stencil calculation. In the embodiment of the present application, the address space corresponding to each process is uniformly addressed to form a unified address space, and the memory address space of multiple processes is mapped to the unified address space. In this way, each process shares the unified address space and can directly access the memory address space corresponding to other processes to read data on other processes. Therefore, the data update delay caused by MPI communication can be avoided, and the computational efficiency of stencil calculation can be improved.

[0042] In the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0043] In the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0044] In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0045] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0046] It is understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It is understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0047] It can be understood that some optional features in the embodiments of the present application may be implemented independently in certain scenarios without relying on other features, such as the solution on which they are currently based, to solve corresponding technical problems and achieve corresponding effects, or may be combined with other features according to needs in certain scenarios. Accordingly, the devices provided in the embodiments of the present application may also realize these features or functions accordingly, which will not be elaborated here.

[0048] In this application, unless otherwise specified, the same or similar parts between the various embodiments can refer to each other. In this application, if there is no special description or logical conflict between the various embodiments, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and different embodiments can be combined to form new embodiments according to their inherent logical relationships. The following implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0049] In order to facilitate understanding of the technical solutions of the embodiments of the present application, a brief introduction to the relevant technologies of the embodiments of the present application is given below:

[0050] Stencil calculation is a cyclic operation mode, which is characterized by traversing the entire data area and performing the same calculation operation on each data in the data area. As the most common calculation mode in many application fields, stencil calculation often occupies a core hotspot position. Its calculation process requires obtaining the data of neighboring grid points according to the numerical format or template to update the data, so it involves a large amount of node communication and data transmission.

[0051] In a specific application scenario, such as in the field of image processing, stencil calculation can be applied to perform image blurring. Specifically, image blurring is typically Gaussian smoothing, which is equivalent to finding the weighted sum of surrounding pixels. Then, the stencil calculation in this scenario is to perform a weighted sum operation of surrounding pixels for all pixel positions on an image, and update each pixel position according to the operation result. It can be understood that Gaussian smoothing of an image can be performed in different regions, and blurring tasks in different regions may belong to different processes. For the pixel positions at the boundary of the region, there will be a part of the surrounding pixels belonging to the first process, which are stored in the memory address space of the first process, and another part of the surrounding pixels belonging to the second process, which are stored in the memory address space of the second process. At this time, when Gaussian smoothing is performed on the pixel positions at the boundary of the region, the two processes need to communicate to obtain the pixel data of all surrounding pixels.

[0052] Specifically, the first process processes the pixel data of the first area of ​​the image to be processed, and the second process processes the pixel data of the second area of ​​the image to be processed... Each area can be divided into grids to form grid data. For example, each grid point in the grid data corresponds to the pixel data of a pixel point. Each grid point can be divided into internal grid points and edge grid points. Figure 1 As shown in the figure, the grid points in the top row, the bottom row, the leftmost row, or the rightmost row of the grid are called edge grid points. The other grid points are internal grid points. This division is because when performing stencil calculation (Gaussian smoothing calculation) on each grid point, the data update of each grid point depends on the corresponding value of the neighboring grid points. Among them, the neighboring grid points of the internal grid points all belong to this area, while the edge grid points have at least one neighboring grid point that belongs to other areas corresponding to other processes.

[0053] The following is a brief introduction to the stencil calculation process for each grid point. When updating each grid point (pixel), you need to first obtain the value of the neighboring grid point corresponding to the grid point. Figure 2As shown, an area is divided into multiple grid points. Among them, the grid points above, below, left and right of grid point 1 are the neighboring grid points 2, 3, 4, and 5 of grid point 1. In a calculation iteration process, it is necessary to determine the update value of grid point 1 based on the initial values ​​of grid point 1 and neighboring grid points 2, 3, 4, and 5, and the update value of grid point 1 obtained by this calculation will be used as the initial value of grid point 1 in the next iteration process. Then after multiple iterations, the final value of grid point 1 is obtained. Similarly, the data iteration process of other grid points is similar to that of grid point 1. For example, in each iteration process, it is necessary to update the data of grid point 2 according to the initial values ​​of neighboring grid points 1, 6, 7, and 8 corresponding to grid point 2, which will not be repeated here. It can be understood that only when all grid points are updated can the next round of data iteration be performed, and the calculation results of each node after the previous data iteration will be determined as the initial value of each node in the next round of data iteration. In a single iteration, since the neighboring grid points of the internal grid points are all within the calculation area, that is, the stencil calculation only requires the data within the current process, and since the data in the same process is stored in the same memory address space, the initial value of each neighboring grid point can be directly obtained according to the memory address corresponding to each neighboring grid point. When updating the edge grid points, it is necessary to communicate with at least one other process to obtain the data of the neighboring grid points in other processes before proceeding.

[0054] Currently, all processes communicate with each other by calling MPI functions. During data communication, the process needs to switch from user mode to kernel mode, and then complete all communications in kernel mode. Therefore, when performing stencil calculations, the calculation of internal grid points and the communication process of edge grid points can overlap, that is, while performing internal grid point calculations, it is possible to communicate with other processes to obtain the values ​​of neighbor grid points on other processes corresponding to each edge grid point. After all communications are completed, that is, after obtaining the initial values ​​of neighbor grid points required by all edge grid points, the process switches from kernel mode to user mode, and then calculates and updates the data corresponding to each edge grid point. For the specific calculation process, please refer to Figure 3 Flowchart shown.

[0055] like Figure 3As shown, when calculating the stencil, first initialize the data of each node. Then call the MPI function for communication. The communication process includes obtaining the data of the neighboring grid points of each edge grid point in other processes, and sending the initial values ​​corresponding to each edge grid point to other processes. While communicating, each internal grid point can be calculated, that is, the internal grid points are updated first. After the internal grid points are updated, it is necessary to call the wait function to check whether the communication is completed, that is, whether the communication corresponding to all edge grid points is completed. When all are completed, calculate and update each edge grid point. After all the internal grid points and edge grid points are updated, enter the next data iteration. It can be understood that the updated value of each grid point in the previous round is the initial value of each grid point in the next round.

[0056] From the above process, we can see that although the calculation and communication process of internal grid points can overlap, the calculation and communication process of edge grid points cannot overlap. Each edge grid point needs to be calculated and updated after all communications are completed. In this way, the calculation of edge grid points still takes a lot of time, causing delays in updating edge grid point data, which seriously affects the efficiency of stencil calculation. At the same time, when calling MPI functions for communication, the process needs to switch between user mode and kernel mode multiple times, which will seriously consume system resources.

[0057] Based on this, an embodiment of the present application provides a data processing method and related equipment based on stencil calculation. In an embodiment of the present application, the address space corresponding to each process is uniformly addressed to form a unified address space. In this way, the second address of the neighbor grid point in the memory address space of other processes can be determined based on the first address of the neighbor grid point in the unified address space, and then the memory address space of other processes can be directly accessed according to the second address to obtain the data of the neighbor grid point. In this way, multiple processes do not need to call MPI to communicate to obtain the data of the neighbor grid point, thereby avoiding data update delays and system resource loss caused by MPI communication. The following is a detailed introduction to the embodiment of the present application.

[0058] Before introducing the embodiments of the present application, the network architecture corresponding to the embodiments of the present application is first introduced:

[0059] Figure 4 A system architecture diagram of a cluster computing system provided in an embodiment of the present application. Figure 4 As shown, the cluster computing system includes multiple computing nodes interconnected by a network, which is essentially a group of computers. Specifically, each node runs its own operating system instance, and all nodes use the same hardware and the same operating system. Among them, all nodes run multiple processes, and each node runs one or more processes. The process is used to process the data of the cluster application. It can be understood that each process needs to communicate to achieve data transmission.

[0060] Based on the above cluster computing system, the operating system running on the cluster computing system is introduced below. Figure 5 This is a system architecture diagram of an operating system provided in an embodiment of the present application. Figure 5 As shown in Figure 1, the kernel is the most basic part of the operating system. It is responsible for managing the processes, memory, device drivers, files, and network systems of the cluster computing system, and determines the performance and stability of the system. The kernel provides secure access to computer hardware for many processes and determines when and how long a program can operate on a certain part of the hardware.

[0061] A process is a running activity of a program in a computer (node) on a certain data set. It is the basic unit for resource allocation in a cluster computing system and the foundation of the operating system structure. Multiple processes can be executed concurrently, and each process is a basic unit that can run independently, and is also an independent unit for resource allocation and scheduling in the system. Since each process is independent and mutually constrained, the execution of the process is intermittent, and each process moves forward at its own independent and unpredictable speed.

[0062] Currently, each process is assigned an independent address space, where the address space is determined by the address bus and registers of the processor corresponding to each node. The memory address corresponding to each process is composed of the starting memory address and the space offset. Within the process, each program shares the address space and can directly access the data in the memory according to the memory address. The address spaces of multiple processes are independent. If a process needs to call data on other processes, it needs to communicate between processes through the kernel. The kernel obtains the memory address corresponding to other processes and then instructs other processes to access the memory address. After obtaining the relevant data, the kernel controls and sends it to the process that issued the data request.

[0063] Specifically, the communication process between processes is that each process switches from user state to kernel state, and then the kernel calls MPI communication. A process sends a data request to a neighbor process through the MPI interface, and then obtains the memory address of the neighbor process. After the neighbor process obtains the memory data through the memory address, the data in the neighbor process is sent to the process. Since the processes exchange data based on MPI communication, generally, all the required data on the neighbor process needs to be obtained. After the MPI communication is completed, it switches from kernel state to user state again to continue the process.

[0064] In the embodiment of the present application, the address space corresponding to each process is uniformly addressed to obtain a unified address space, so that all processes share the unified address space, and then, it is necessary to establish a mapping relationship between the memory space address corresponding to each process and the unified address space. Among them, the virtual memory address corresponding to the unified address space is based on a unified starting memory address and a space offset. Exemplarily, the memory address space of process 1 can correspond to 0 to 100 in the unified address space, the memory address space of process 2 corresponds to 101 to 200 in the unified address space, and the memory address space of process 3 can correspond to 201 to 300 in the unified address space... In this way, all processes share the unified address space, and can directly obtain the first address of a certain data on other processes in the unified address space, and then obtain the second address of the data in the memory address space of a certain process according to the mapping relationship between the first address and each memory address space and the unified address space, and then directly access the corresponding memory address address according to the second address to obtain the memory data. In this way, the traditional MPI communication process can be omitted.

[0065] Based on the above architecture, the new communication process between processes is introduced in detail below. Figure 6 A schematic diagram of a data processing method based on stencil calculation provided in an embodiment of the present application. Figure 6 As shown, the data processing method includes the following steps:

[0066] 601. Address spaces of multiple processes are uniformly addressed to form a unified address space, and the corresponding relationship between the unified address space and the memory address spaces of multiple processes is recorded.

[0067] In a cluster computing system, multiple processes are executed in parallel to jointly process data in cluster applications. Each process processes multiple data, and each process is allocated an independent memory address space, and the data to be processed by each process is stored in its own memory.

[0068] In order to allow processes to exchange data without using traditional MPI communication, in this embodiment, the address spaces of multiple processes are first uniformly addressed to form a virtual unified address space. Then the memory address space corresponding to each process is mapped to the unified address space, and the correspondence between the unified address space and the memory address spaces of multiple processes is recorded. In this way, it is equivalent to all processes sharing a unified address space. When a process wants to obtain data on another process, it only needs to obtain the address in the memory address space actually corresponding to the data based on the address of the data in the unified address space, and then it can directly access the original memory address space based on the address in the memory address space to obtain the data.

[0069] The following is an introduction to the unified compilation process:

[0070] For example, the data of the cluster application is data arranged in a certain order, and the size of each data is the same. Then, when the address spaces of multiple processes are uniformly addressed, the first address of the first data in the data of the cluster application is first used as the first address of the unified address space finally formed, and the first address is synchronized to each process. Then, the memory address space corresponding to each process can be programmed in sequence according to the first address to obtain a unified virtual address. Then, the corresponding relationship between the virtual address and the original memory address can be established and recorded.

[0071] 602. In response to a command of a first process among multiple processes to obtain first data of a second process, obtain a first address of the first data in a unified address space according to information of the first data.

[0072] When multiple processes need to communicate, a data request command for first data belonging to another process sent by one process can be responded to to determine the first address of the first data in the unified address space. Specifically, the order of the first data in the data of the cluster application is first determined, and then the address offset is determined according to the order of the first data in the data of the cluster application and the size of each data, and then the first address of the first data in the unified address space is determined based on the first address and the address offset of the unified address space.

[0073] 603. Obtain, according to the first address, a second address in the memory address space of the second process corresponding to the first address.

[0074] Then, it is necessary to determine the second address in a certain memory address space corresponding to the above-mentioned first address based on the correspondence between the unified address space and the memory address spaces of multiple processes. It can be understood that the second address is the address of the memory space where the first data is actually stored.

[0075] 604. Read the first data from the memory address space of the second process according to the second address.

[0076] After obtaining the second address of the first data in the actual memory address space, the process can directly access the actual memory address space according to the second address to obtain the first address to be read.

[0077] In an embodiment of the present application, when multiple processes jointly process the data of a cluster application, the memory address spaces of the multiple processes are first uniformly addressed to form a virtual unified address space, and then the mapping relationship between each memory address space and the unified address space is recorded. Next, when multiple processes communicate, first respond to an instruction for a process to obtain data on another process, determine the first address of the required data corresponding to the instruction in the unified address space, and then query the mapping relationship between each memory address space and the unified address space, determine the second address of the required data in the memory address space of another process based on the first address of the required data in the unified address space, and finally query the memory address space of another process according to the direct second address to read the required data. Since the memory address space corresponding to each process is uniformly addressed and the unified address space is shared, when the processes communicate to exchange data, they can directly access the memory address space of other processes according to the address of the data in the unified address space, without calling MPI communication to obtain data. In this way, when a process performs data calculation based on the data on other processes, it does not need to wait for all communications to be completed before starting the calculation, but can achieve the overlap of communication and calculation, that is, calculation while communicating. Therefore, the embodiment of the present application can avoid the processing delay caused by MPI communication and improve the computing efficiency of data calculation. At the same time, it can also avoid multiple switching between the user state and the kernel state of the process caused by calling the MPI function for communication, thereby saving system resources and reducing the impact on the system.

[0078] It can be understood that in the above-mentioned image processing application scenario, stencil calculation of the image to be processed involves communication between processes. Figure 6 The method provided in the illustrated embodiment is to obtain the pixel value of the neighboring pixel points on other processes corresponding to the edge pixel points. Similarly, if the data processed by one process is mapped to form the first grid data, and the data processed by another process is mapped to form the second grid data, and when the stencil calculation is performed on the edge grid points of each grid data, it is necessary to obtain the data of the neighboring grid points on other processes of the edge grid points, the data method can also be used to obtain the data of the neighboring grid points. It can be understood that Figure 6 The illustrated embodiment describes the steps of obtaining the values ​​of the neighboring grid points corresponding to the edge grid points. The calculation process of the edge grid points will be introduced below.

[0079] Figure 7 The flowchart of a data updating method based on stencil calculation provided in the embodiment of the present application is as follows. It can be understood that the present embodiment is used to describe the process of single data iteration of each grid point in the grid data. Figure 7 As shown, the data updating method includes the following steps:

[0080] 701. Obtain initial values ​​of edge grid points in first grid data.

[0081] When updating the edge grid points of the first grid data corresponding to the first process, the initial value of each edge grid point for this data iteration must first be obtained. Each process corresponds to a memory area, which is used to store the data corresponding to each grid point in each process. Specifically, the memory area can be accessed according to the memory address corresponding to each grid point to obtain the initial value corresponding to each edge grid point.

[0082] 702. Determine a first neighboring grid point in the second grid data corresponding to the edge grid point.

[0083] Then, determine the neighboring grid points corresponding to each edge grid point. Each grid point in each grid data is distributed in two dimensions, and each grid point corresponds to four neighboring grid points above, below, left and right. Each neighboring grid point of the internal grid point is inside the first grid data, while the edge grid point may have neighboring grid points of the second grid data corresponding to other processes. For example, Figure 8 As shown, a neighbor grid point 7011 of edge grid point 6011 belongs to the second grid data corresponding to other processes, while neighbor grid points 6012, 6013, and 6014 are located in the first grid data corresponding to this process. Edge grid point 6016 has two neighbor grid points 8011 and 9011 that belong to other processes, and only two neighbor grid points 6019 and 6020 are located in the first network data corresponding to this process. Therefore, when updating edge grid point 6011, it is necessary to determine the neighbor grid point 7011 on other processes and obtain the value of neighbor grid point 7011 from other processes. Similarly, when updating edge grid point 6016, it is necessary to determine the values ​​of neighbor grid points 8011 and 9011 from two neighbor processes respectively.

[0084] 703. Determine a second address of the first neighbor grid point in the memory address space according to a first address of the first neighbor grid point in the unified address space, and obtain an initial value of the first neighbor grid point according to the second address.

[0085] In this embodiment, the address spaces corresponding to multiple processes are uniformly compiled to form a shared virtual unified address space. Then, the process can know the first address of the first neighbor grid point on the neighbor process in the unified address space. Then, based on the mapping relationship between the unified address space and the memory address space of each process, the second address of the first address in the memory address space of the neighbor process can be obtained, so that the memory address space of the neighbor process can be directly accessed according to the second address. There is no need to communicate by calling MPI functions. It can be understood that in the stage of data initialization, the memory address of the neighbor grid points belonging to other processes corresponding to each edge grid point can be obtained. When a certain edge grid point needs to be updated, the memory area corresponding to other processes can be directly accessed according to the memory address of the neighbor grid point to obtain the value corresponding to the neighbor grid point.

[0086] 704. Update data corresponding to the edge grid point according to the initial value of the edge grid point and the initial value of the first neighboring grid point.

[0087] After obtaining the initial values ​​of all neighboring grid points corresponding to the edge grid points, the update value of the edge grid points can be determined based on the initial value of the edge grid points, the initial values ​​of the neighboring grid points in the first calculation area, and the initial values ​​of the neighboring grid points in other processes, and the data corresponding to the edge grid points can be updated. It can be understood that the updated data is the initial value of the next iteration. And each edge grid point needs to be updated, and after all grid points in the first calculation area are updated, the next round of updates can be entered.

[0088] In the embodiment of the present application, the memory area corresponding to each process is uniformly addressed, so that the corresponding memory area on other processes can be directly accessed based on the unified memory address on any process to obtain the data on other processes. Therefore, there is no need to call MPI for communication between processes. In this way, when performing stencil calculations, the neighboring grids of edge grids on other processes can be directly accessed to obtain the corresponding data to update the data of the edge grids. In this way, the MPI communication in the stencil calculation process is omitted, and the overlap of communication and calculation can be achieved between the edge grids, that is, the situation that the calculation is performed for other edge grids while communicating for some edge grids can be achieved. It is no longer necessary to wait for all communications to end before calculating the edge grids. Therefore, the data update delay caused by MPI communication can be avoided, and the calculation efficiency of stencil calculation can be improved. At the same time, it can also avoid the multiple switching of user state and kernel state of the process due to calling MPI functions for communication, thereby saving system resources and reducing the impact of stencil calculation on the system.

[0089] It is understandable that the above embodiment introduces the updating of edge grid points in a single data iteration process. The following describes in detail the single data iteration process of stencil calculation in combination with internal grid points. Fig. 9 A flow chart of another data updating method based on stencil calculation provided in an embodiment of the present application. Fig. 9 As shown, the data updating method includes the following steps:

[0090] 901. Determine neighboring grid points corresponding to each internal grid point in the first grid point data respectively.

[0091] When performing stencil calculation on the first grid point data corresponding to the first process, the internal grid points are updated first, and then the edge grid points are updated. Since the internal grid point update does not require data belonging to other processes, there is no need to communicate with other processes. After all the internal grid points in the first calculation area are updated, communication and edge grid point calculation are performed. Specifically, for an internal grid point, it is necessary to obtain the initial values ​​of the neighboring grid points in the four directions of up, down, left, and right corresponding to the internal grid point, and then update the internal grid point according to the initial values ​​of all neighboring grid points.

[0092] 902. Update data corresponding to each internal grid point according to the initial value of the neighboring grid point corresponding to each internal grid point.

[0093] Specifically, it is necessary to execute the same calculation instruction for each internal grid point, and update all the internal grid points using the initial values ​​of the neighboring grid points corresponding to each internal grid point.

[0094] 903. When all data corresponding to the internal grid points are updated, the data corresponding to the edge grid points are updated.

[0095] After all the internal grid points are updated, communicate with other processes to update the data corresponding to the edge grid points. It can be understood that the embodiment of the present application is no longer based on MPI communication to obtain data on other processes, but is based on unified memory addressing. It can directly obtain the second address of the neighbor grid point in the memory address space of other processes based on the first address of the neighbor grid point on other processes in the unified address space. Then directly access the memory area corresponding to other processes based on the second address, and directly access the unified memory address to obtain the value of the neighbor grid point on other processes. For the specific process, please refer to Figure 6The process shown in the illustrated embodiment will not be described in detail here. It can be understood that when updating each edge grid point, the update of each edge grid point is isolated. That is, for one edge grid point, calculation can be performed immediately after the communication is completed. For all edge grid points, communication and calculation can be performed synchronously, without waiting for all communications to be completed before calculating the edge grid points, which will greatly reduce the update delay of the edge grid points.

[0096] 904. In response to the data query request for the edge grid points sent by the second process, send the initial values ​​of the edge grid points to the second process.

[0097] It is understandable that when the calculation area corresponding to this process is updating data, the edge grid points of other processes must also obtain the initial values ​​of the neighboring grid points from this process, that is, the transmission during the communication process. Specifically, when a data query request (data access request) sent by the second process is received, it is necessary to first determine whether the update of the internal grid points is completed. That is, after the internal grid point update is completed, communication and edge grid point update are performed. Among them, communication and edge grid point calculation overlap, that is, while sending the initial value of an edge grid point to the second process, another edge grid point can be calculated. If the data of the edge grid point to be requested is being calculated, the calculation instruction is executed first, and the initial value of the edge grid point is sent to the second process after the update is completed.

[0098] For example, the initial values ​​of multiple edge grid points can also be sent in a centralized manner, that is, the edge grid points are divided into multiple groups. When the initial value of a certain edge grid point needs to be sent to other processes during the communication process, it is first determined whether a group of edge grid points corresponding to the edge grid point has been completely updated. After all the edge grid points in the group have been completely updated, the initial values ​​of the edge grid points are sent to different processes in a centralized manner in response to data access requests. This can improve communication efficiency.

[0099] For example, the number of edge grid points in each group, ie, the preset number of grid points, can be determined according to actual computing requirements, and then all edge grid points are grouped according to the preset number of grid points for centralized communication.

[0100] In the above embodiment, when a calculation area is performing stencil calculation, in each data iteration process, the internal grid points of the calculation area are updated first, and then the edge grid points are updated. In the process of updating the edge grid points, communication and calculation are overlapped. Based on the unified addressing of the memory area corresponding to each process, the neighbor grid points of the edge grid points in other processes can be directly accessed according to the unified memory address, and the corresponding data can be obtained to update the edge grid points, thereby eliminating the MPI communication process.

[0101] Based on the above embodiments, Fig.10This is a flow chart of another data updating method based on stencil calculation provided in an embodiment of the present application. It can be understood that this embodiment introduces the entire data iteration process. Fig.10 As shown, the method comprises the following steps:

[0102] 1001. Initialize data.

[0103] First, initialize the data of each grid point in the grid point data corresponding to each process, and then enter the data iteration process. It can be understood that the memory area corresponding to each process is uniformly addressed, so in the initialization stage, the memory address of the neighbor grid points of each edge grid point in other processes can be directly obtained.

[0104] 1002. Update the internal grid points.

[0105] When updating data, the internal grid points are updated first. All neighboring grid points of the internal grid points are located inside the calculation area. Therefore, the values ​​of the neighboring grid points are directly obtained and the internal grid points are updated according to the values ​​of the neighboring grid points.

[0106] 1003. Determine whether all internal grid points are updated. If yes, execute step 904; if no, continue to wait.

[0107] It can be understood that when updating data, the internal grid points are updated first, and after all the internal grid points are updated, communication and calculation of edge grid points are performed.

[0108] 1004. Update the edge grid points.

[0109] After the internal grid points are updated, it is necessary to communicate with other processes to obtain the values ​​of the neighbor grid points of the edge grid points on other processes. In this application, the communication between processes no longer relies on MPI communication, but directly accesses the data on other processes based on a unified memory address. Specifically, it is necessary to obtain the initial values ​​of the neighbor grid points inside the calculation area and the initial values ​​of the neighbor grid points on other processes, and update the edge grid points according to the initial values ​​of all neighbor grid points to obtain the updated values.

[0110] 1005. Determine a residual value according to the updated data of the edge grid points and the initial value of the edge grid points.

[0111] After the update of each grid point is completed, it is necessary to determine the difference between the updated value and the initial value, determine the residual value based on the updated value and the initial value, and see whether the residual converges.

[0112] 1006. Determine whether the residual value is greater than a preset threshold. If so, update again. If not, end the data update process.

[0113] Specifically, it is to determine whether the residual value is greater than a preset threshold. If it is greater than, it means that the residual has not converged. If it is less than, it means that the residual has converged. If it has not converged, it is necessary to enter the next round of iteration and update again. It can be understood that the updated value of each grid point in the previous round will be the initial value of the next round of iteration. When the residual converges, it is necessary to end the entire process and complete the data update.

[0114] Based on the above description, in a specific example, for the communication of edge grid points in the calculation area, under unified addressing, the intrinsic function can be abstracted. When the edge grid points need to be updated, the intrinsic function can be called to directly perform stencil calculations.

[0115] For example, assuming that data a and data b are neighboring grid points on different processes, first, the memory addresses saddr and daddr of data a and b need to be obtained;

[0116] Then you need to add a and b to the corresponding registers sregister and dregister to implement the addition, subtraction, multiplication and addition calculations required in the stencil.

[0117] For example, you can customize the stencil data addition function stencilfadd; variables (float64x1_t a, float64x1_t b) to implement

[0118] mov sregister,saddr

[0119] mov dregister,daddr

[0120] fadd daddr,sregister,dregister

[0121] For example, you can customize the stencil data subtraction function stencilfsub; variables (float64x1_t a, float64x1_t b) to achieve

[0122] mov sregister,saddr

[0123] mov dregister,daddr

[0124] fsub daddr,sregister,dregister

[0125] For example, you can customize the stencil data multiplication and addition function stencilfma; variables (float64x1_t a, float64x1_t c, float64x1_t b) to implement

[0126] mov sregister,saddr

[0127] mov cregister,caddr

[0128] mov dregister,daddr

[0129] fma daddr,sregister,cregister,dregister

[0130] The embodiment of the present application provides a data updating device based on stencil calculation. In the embodiment of the present application, the functional modules of the device can be divided according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0131] In the case of dividing each functional module into corresponding functional modules, Fig.11 A possible structural schematic diagram of a data processing device based on stencil calculation involved in the above embodiment is shown. Fig.11 As shown, the data processing device is applied to a cluster computing system. The cluster computing system includes multiple processes, and the multiple processes are used to process data of cluster applications. Each process processes multiple data, and each process is allocated a memory address space. The data processing device includes:

[0132] The processing unit 1101 is used to uniformly address the address spaces of multiple processes to form a unified address space, and record the corresponding relationship between the unified address space and the memory address spaces of multiple processes.

[0133] The acquiring unit 1102 is configured to respond to a command from a first process among multiple processes to acquire first data of a second process, and acquire a first address of the first data in a unified address space according to information of the first data.

[0134] The acquiring unit 1102 is further configured to acquire, according to the first address, a second address in the memory address space of the second process corresponding to the first address.

[0135] The reading unit 1103 is used to read the first data from the memory address space of the second process according to the second address.

[0136] In an optional implementation, the data of the cluster application is data arranged in a certain order, and each data has the same size. The data updating device further includes a sending unit 1104 .

[0137] The sending unit 1104 is configured to send the first address of the first data in the data of the cluster application to the multiple processes, so that each of the multiple processes uses the first address as the first address of the unified address.

[0138] In an optional implementation, the information of the first data is the order of the first data in the data of the cluster application.

[0139] The acquiring unit 1102 is specifically configured to determine the first address according to the sequence of the first data in the data of the cluster application, the first address of the unified address space, and the size of each data.

[0140] In an optional implementation, the acquiring unit 1102 is specifically configured to determine the second address according to the first address and a correspondence between the unified address space and the memory address spaces of the multiple processes.

[0141] In an optional embodiment, the plurality of data processed by the first process forms grid data, wherein each data in the plurality of data processed by the first process corresponds to a grid point in the grid data, and the first data is data in the grid data at the top row, the bottom row, the leftmost row, or the rightmost row of grid points.

[0142] Each module of the above-mentioned device can also be used to perform other actions in the above-mentioned method embodiment. All relevant contents of each step involved in the above-mentioned method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0143] Fig.12 It is a structural diagram of a processing device provided in an embodiment of the present application. The processing device 1200 may include one or more central processing units (CPU) 1001 and a memory 1205. The memory 1205 stores one or more application programs or data.

[0144] The memory 1205 may be a volatile storage or a persistent storage. The program stored in the memory 1205 may include one or more modules, each of which may include a series of instruction operations in the management server. Furthermore, the central processor 1201 may be configured to communicate with the memory 1205 and execute a series of instruction operations in the memory 1205 on the processing device 1200.

[0145] The central processor 1201 is used to execute the computer program in the memory 1205, so that the processing device 1200 is used to uniformly address the address spaces of multiple processes to form a unified address space, and record the corresponding relationship between the unified address space and the memory address spaces of multiple processes; respond to the command of the first process in multiple processes to obtain the first data of the second process, obtain the first address of the first data in the unified address space according to the information of the first data; obtain the second address in the memory address space of the second process corresponding to the first address according to the first address; read the first data from the memory address space of the second process according to the second address. For specific implementation methods, please refer to Figure 6 The steps shown in the illustrated embodiment will not be repeated here.

[0146] The processing device 1200 may also include one or more power supplies 1202, one or more wired or wireless network interfaces 1203, one or more input and output interfaces 1204, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0147] The processing device 1200 may execute the aforementioned Figures 6 to 10 The operations performed in the illustrated embodiment will not be described in detail here.

[0148] The present application also provides a chip, which includes: a processor and an interface circuit; the interface circuit is used to receive code instructions and transmit them to the processor; the processor is used to run the code instructions to execute Figures 6 to 10 Provides any data processing method and data updating method based on stencil calculation.

[0149] The present application also provides a computer-readable storage medium in which at least one computer program instruction is stored. The computer program instruction is loaded and executed by a processor to implement the above Figures 6 to 10 Provides any data processing method and data updating method based on stencil calculation.

[0150] The present application also provides a computer program product, including a computer-executable instruction, which, when executed on a computer, causes the computer to execute Figures 6 to 10 Provides any data processing method and data updating method based on stencil calculation.

[0151] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, all or part of the processes (or functions) described in the embodiments of the present application are implemented. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with the medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)), etc. In the embodiment of the present application, the computer may include the aforementioned device.

[0152] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in a claim. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

Claims

1. A data processing method based on stencil calculation, applied to a cluster computing system, characterized in that: The cluster computing system includes a plurality of processes, and the plurality of processes are used to process data of cluster applications; wherein each process processes a plurality of data, and each process is allocated a memory address space, and the method includes: Performing unified addressing on the address spaces of the multiple processes to form a unified address space, and recording a corresponding relationship between the unified address space and the memory address spaces of the multiple processes; In response to a command of a first process among the multiple processes to obtain first data of a second process, obtaining a first address of the first data in the unified address space according to information of the first data; Acquire, according to the first address, a second address in the memory address space of the second process corresponding to the first address; The first data is read from the memory address space of the second process according to the second address.

2. The method according to claim 1, characterized in that The data of the cluster application is data arranged in a certain order, and the size of each data is the same; The step of uniformly addressing the address spaces of the multiple processes to form a unified address space includes: The first address of the first data in the data of the cluster application is sent to the multiple processes, so that each of the multiple processes uses the first address as the first address of the unified address space.

3. The method according to claim 2, characterized in that The information of the first data is the order of the first data in the data of the cluster application, and obtaining the first address of the first data in the unified address space according to the information of the first data includes: The first address is determined according to the order of the first data in the data of the cluster application, the first address of the unified address space, and the size of each data.

4. The method according to claim 3, characterized in that The acquiring, according to the first address, a second address in the memory address space of the second process corresponding to the first address includes: The second address is determined according to the first address and the correspondence between the unified address space and the memory address spaces of the multiple processes.

5. The method according to claim 3, characterized in that: The multiple data processed by the first process form grid data; wherein each data among the multiple data processed by the first process corresponds to a grid point in the grid data; the first data is the data of the grid points in the top row, the bottom row, the leftmost row, or the rightmost row of the grid in the grid data.

6. A data processing device based on stencil calculation, applied to a cluster computing system, characterized in that: The cluster computing system includes a plurality of processes, and the plurality of processes are used to process data of cluster applications; wherein each process processes a plurality of data, and each process is allocated a memory address space, and the data processing device includes: A processing unit, configured to uniformly address the address spaces of the multiple processes to form a unified address space, and record a correspondence between the unified address space and the memory address spaces of the multiple processes; An acquiring unit, configured to respond to a command from a first process among the multiple processes to acquire first data of a second process, and acquire a first address of the first data in the unified address space according to information of the first data; The acquiring unit is further configured to acquire, according to the first address, a second address in the memory address space of the second process corresponding to the first address; A reading unit is used to read the first data from the memory address space of the second process according to the second address.

7. The data processing device according to claim 6, characterized in that: The data of the cluster application is data arranged in a certain order, and the size of each data is the same; the data processing device also includes a sending unit; The sending unit is used to send the first address of the first data in the data of the cluster application to the multiple processes, so that each of the multiple processes uses the first address as the first address of the unified address.

8. The data processing device according to claim 7, characterized in that: The information of the first data is the order of the first data in the data of the cluster application. The acquisition unit is specifically configured to determine the first address according to the order of the first data in the data of the cluster application, the first address of the unified address space, and the size of each data.

9. The data processing device according to claim 8, characterized in that: The acquisition unit is specifically configured to determine the second address according to the first address and a correspondence between the unified address space and the memory address spaces of the multiple processes.

10. The data processing device according to claim 8, characterized in that: The multiple data processed by the first process form grid data; wherein each data among the multiple data processed by the first process corresponds to a grid point in the grid data; the first data is the data of the grid points in the top row, the bottom row, the leftmost row, or the rightmost row of the grid in the grid data.

11. A processing device, characterized in that: The processing device comprises a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor to enable the processing device to implement the method according to any one of claims 1-5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program instruction, and the computer program instruction is loaded and executed by a processor to implement the method according to any one of claims 1 to 5.

Images