Programmable data exchange system

Abstract

The invention discloses a programmable data exchange system which comprises the following steps: on a data use side, expanding a programming language of the data use side to allow a user to write an access memory in a source code; providing side services are deployed on the data providing side; the service is launched by the provider-side runtime and allows the user to modify by customizing the plug-in. By using the method, a part of data access-intensive program codes, namely the memory access sub, are packaged and sent to the far-end system for execution, and data preprocessing work such as filtering, compression, coding and flow pushing can be completed on the far-end system according to application requirements, so that the data volume needing to be exchanged between systems on two sides and the number of interaction times are reduced, and the data processing efficiency is improved. And the overall application performance is greatly improved. The method can be widely applied to the field of computer systems.

Description

Technical Field

[0001] The present invention relates to the field of computer systems, and in particular to a programmable data exchange system. Background Art

[0002] The computing network aims to connect the computing power supply, application development, operation services, users and other capabilities and resources in the industrial ecology, operate supercomputing centers with the Internet mindset, and build an integrated computing network and service platform. It not only forms an efficient data transmission network between computing centers, but also builds and improves a cross-domain integrated computing power scheduling network and an ecological collaboration network for diverse applications, connecting supply and demand, expanding applications, and prospering the ecology.

[0003] Among them, data exchange between computing power systems is the basic key technology supporting the computing power Internet ecosystem. Unlike general personal computer systems, information exchange between computing power systems has the characteristics of large data volume and high parallelism, and many calculations are time-sensitive, which places extremely high demands on the bandwidth and latency of the Internet. On the other hand, the computing power network has strong endpoint computing power that ordinary computer networks do not have, so in many application scenarios, the amount of data that must be transmitted can be reduced by placing part of the calculation near the data end, which can greatly reduce the demand for network resources and improve the overall computing efficiency of the application.

[0004] However, the software and hardware environments of various computing systems are currently very different, and there is a lack of a unified data operation interface, which poses a severe programming challenge to application development. At the same time, the data exchange mode across computing systems often depends on the specific field application, so there is an opportunity to significantly optimize network data transmission at the application level. Finally, for data providers, since the needs of each user and application are often different, a fixed data access interface cannot adapt to diverse operation logic. Summary of the invention

[0005] In view of this, in order to solve the serious redundant data transmission and additional operation request problems caused by the incomplete matching between the interface and the requirements in the existing switching system, the present invention proposes a programmable data switching system, the system comprising:

[0006] On the data user side, the programming language of the data user side is extended to allow users to write access sub-programs in the source code;

[0007] Memory accessors are small pieces of code that are closely related to data access and are defined in the form of functions (subroutines). They run independently of location and can be sent to remote computing systems for execution. They include message memory accessors and streaming memory accessors. The former transmits complete message packets and exits after the memory accessor runs, exchanging data in a request-response mode; the latter can run continuously on the remote system and continuously exchange message packets through the streaming channel.

[0008] On the data provider side, a provider service is deployed; this service is started by the provider runtime and allows users to modify it through custom plug-ins. The data provider service program can work under ordinary user permissions without the intervention of the computing center administrator, so it has the ability to be directly deployed on the current computing power systems.

[0009] On the data usage side, the application works in MPI multi-process mode, and on the remote data provider side, the memory access sub-system also runs in parallel in MPI multi-process mode. This setting solves the data transmission problem in this many-to-many situation, shields the application developers from the underlying details, and greatly simplifies the writing of application code.

[0010] Based on the above scheme, the present invention provides a programmable data exchange system, which encapsulates a portion of data access-intensive program code (referred to as "memory access sub-code" in the present invention) on the data use side and sends it to the remote system for execution. The memory access sub-code is written by the application developer, and can complete data pre-processing such as filtering, compression, encoding, and streaming on the remote system according to application requirements, thereby reducing the amount of data that needs to be exchanged between the two systems and the number of interactions, greatly improving the overall performance of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a block diagram of the architecture of a programmable data exchange system of the present invention;

[0012] Figure 2 This is a code schematic diagram of the present invention taking a C language program pseudo code including a memory access subroutine as an example;

[0013] Figure 3 It is a schematic diagram of the code of the present invention for screening the even-numbered columns of a certain file and transposing and returning them. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0016] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0017] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0018] Reference Figure 1 , is an optional example of the architecture block diagram of the programmable data exchange system proposed by the present invention, Figure 1 The overall workflow of the system is explained as follows:

[0019] First, the user should deploy the provider-side service of the system of the present invention on the computing power system where the data is located. The service is started by the provider-side runtime in the present invention, and allows users to modify it through customized plug-ins. For example, the system can be designed to provide two plug-ins, POSIX and ODBC, which are used to support file system access and SQL database access on the computing power system respectively. Users can adjust these two plug-ins through configuration files to manage the real data resources they access at the bottom. For other types of data resources, users can also create custom plug-ins through the provider-side development kit in the present invention. When building a custom plug-in, the user needs to first write the interface definition, then use the development kit to generate the interface declaration file and the framework code of the plug-in, then write the plug-in function code based on the framework code, and finally compile and build the plug-in and place it in the provider-side runtime.

[0020] Next, on the data usage side, the present invention extends the programming language C / C++ / Fortran on the data usage side, and the extended language allows users to write memory access subroutines in the source code. The user-side development kit in the present invention can separate the source code developed using the extended language into memory access subroutines and ordinary source code programs during compilation. The former is compiled using a dedicated compiler (invisible to the user); the latter is compiled using an ordinary compiler, and then the two are merged into a single application through a linking process. For custom plug-ins on the data provider side, the corresponding interface declaration files must be introduced during compilation and construction. The functions of the custom plug-in cannot be used when the declaration is missing, but other existing functions are not affected. The operation and use of the program are entirely determined by the application, but its operation and startup depend on the runtime support on the user side of the present invention.

[0021] Finally, during the working process of the application, the data exchange process of the two systems is completely taken over by the user-side runtime. The user-side runtime will connect to the provider-side runtime to jointly negotiate the operation of the memory access subroutine and complete the data exchange. Usually, the memory access subroutine will be sent to the remote data provider system for operation, but in special cases (such as the remote computing system is overloaded or resource-constrained), the memory access subroutine can also run locally. This decision-making process is completely determined by the runtime system and is invisible to applications and developers.

[0022] This embodiment allows developers to freely write access and operation logic for remote data by stripping some data access-intensive codes into independent functions that can be run remotely, so as to independently select the data to be exchanged and the encoding method. Among them, the memory access sub-function can be freely programmed to combine the service interface provided by the remote end in any computable way, which solves the problem that the fixed access interface in various previous methods cannot fully meet the application requirements. Through reasonable programming and division of logic, application developers can completely eliminate redundant data transmission and remote interaction.

[0023] In some feasible embodiments, a programming method and a programming model for data usage side access memory are also provided. Figure 2 , taking a C language program pseudo code containing a memory access subroutine as an example, a message memory access subroutine cnet_function and a common C language function host_function are given. The former will be compiled into a memory access subroutine by the tool chain of the present invention and will be run locally or remotely at runtime; the latter will be compiled like a common C language code and work on the local system, but because host_function uses cnet_function, the startup and operation of the program must load the runtime support function of the present invention.

[0024] In this embodiment, the code has the following key constraints in writing:

[0025] 1. The __cnet__ keyword must be appended before global identifiers such as functions, types, and global variables of access submodules;

[0026] 2. The __host__ identifier can be added before the local function to distinguish it, but it can also be omitted by default;

[0027] 3. Identifiers with __cnet__ attached can only reference other identifiers with __cnet__ attached. Similarly, identifiers marked or implicitly marked as __host__ can only reference other identifiers with the same tag. Based on this, the two program codes can be strictly distinguished statically at compile time.

[0028] 4. The function signature of the memory accessor is determined by the type of memory accessor. Its parameters and return values ​​are strictly predefined and fixed. The above pseudo code shows a message memory accessor (cnet_mm). In addition, there is a stream memory accessor, whose function signature is as follows: void__cnet__cnet_ss(CNET_Stream*send, CNET_Stream*recv);

[0029] 5. The writing of the memory access subfunction body is similar to that of an ordinary function. Any operation logic can be executed inside the function body, but only the __cnet__ data structure (for example, cnet_Message is different from CNET_Message) and predefined external functions can be used;

[0030] 6. Local functions can only use defined accessors through specified APIs (such as CNET_message in the above code), and different accessor types need to be used with different functions;

[0031] 7. The API has been mixed with the MPI-based parallel programming model, such as the second line of host_function MPI_COMM_WORLD and the fourth line n (specifying the order of remote operation of the memory access sub-end), which realizes the native support for high-performance parallel computing systems;

[0032] In addition, this embodiment also provides relevant code declarations.

[0033] Table 1 Key access sub-API declarations and function descriptions

[0034]

[0035]

[0036] Table 2 Key runtime API declarations and function descriptions

[0037]

[0038] The present invention also provides a code example. Assume that a matrix file " / matrix" stored in rows is provided on the data providing system "remote". The code for using the present invention to filter out the even columns of the file and transpose the returned columns is as follows: Figure 3 As shown in the figure, compared with direct file transmission, the present invention saves about half of the data transmission volume, and compared with the RPC service based on the continuous area matrix, the present invention can reduce at least m / 2 remote interactions. At the same time, the storage child can read files in parallel at the remote end (the parallelism in the case is 10), which greatly improves the performance of data acquisition.

[0039] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A programmable data exchange system, characterized in that: It includes the data usage side and the data provision side, where: The data use side allows the user to write access sub-programs in the source code; The memory access sub-program code is a storage IO-intensive program code, which is remotely executed during the secure runtime of the data provider side.

2. A programmable data exchange system according to claim 1, characterized in that: The provider side service allows users to modify it through custom plug-ins.

3. A programmable data exchange system according to claim 1, characterized in that: Also includes: Write interface definitions and generate interface declaration files and plugin framework code; Writing plug-in function code based on the framework code of the plug-in to obtain a custom plug-in; When the data providing side is running, the custom plug-in is placed.

4. A programmable data exchange system according to claim 1, characterized in that: The memory access sub-module includes a message memory access sub-module and a stream memory access sub-module, wherein: The message access subroutine transmits a complete message message, exchanges data in a request-response mode, and exits after running; The streaming memory access subroutine continues to run at the remote end and continuously exchanges message packets through the streaming channel.

5. A programmable data exchange system according to claim 1, characterized in that: During the operation of the application, the data exchange process of the programmable data exchange system is completely taken over by the data user runtime, which connects to the provider runtime to jointly negotiate the operation of the access sub-device and complete the data exchange.

6. A programmable data exchange system according to claim 1, characterized in that: Also includes: In response to a system decision, the accessor runs locally.

7. A programmable data exchange system according to claim 1, characterized in that: Also includes: On the data usage side, the application program works in MPI multi-process mode; On the data providing side, the memory accessors run in parallel in MPI multi-process mode.

8. A programmable data exchange system according to claim 1, characterized in that: Also includes: Prepends the specified keyword before the global identifier.

9. A programmable data exchange system according to claim 8, characterized in that: Also includes: When writing code, references are restricted to identical identifiers.

Images