A programmable data exchange system

Abstract

The application discloses a programmable data exchange system, which comprises the following parts: on the data usage side, the programming language of the data usage side is extended to allow users to write memory access sub in the source code; the data providing side is provided with a providing side service; the service is started by the providing side runtime and is allowed to be modified by a customized plug-in. By using the application, a part of data access intensive program code, i.e. a memory access sub, is encapsulated and sent to a remote system for execution, filtering, compression, encoding, stream pushing and other data preprocessing work can be completed on the remote system according to application requirements, thereby reducing the data amount and interaction times required to be exchanged between the two systems, and greatly improving the overall performance of the application. The application can be widely applied in the field of computer systems.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer systems, and in particular to a programmable data exchange system. BACKGROUND

[0002] The computing power network aims to connect the capabilities and resources of the supply of computing power, application development, operation service, users and other parties in the industry ecosystem, and to operate supercomputing centers with the thinking of the Internet, build an integrated computing power network and service platform. It not only forms an efficient data transmission network between various computing power centers, but also builds and perfects a cross-domain integrated computing power scheduling network and an ecological cooperation network for various applications, connecting supply and demand, expanding applications, and flourishing the ecosystem.

[0003] Among them, the data exchange between computing power systems is a basic and key technology to support the computing power Internet ecosystem. Unlike general personal computer systems, the exchange of information between computing power systems has the characteristics of large data volume and high parallelism, and many calculations are time-sensitive, so it puts forward very high requirements for the bandwidth and latency of the Internet. On the other hand, the computing power network has strong end-point computing power that ordinary computer networks do not have, so in many application scenarios, a part of the calculation can be placed near the data end to reduce the amount of data that must be transmitted, which can greatly reduce the demand for network resources and improve the overall computing efficiency of the application.

[0004] However, the software and hardware environments of various computing power systems differ greatly, and there is a lack of unified data operation interface, which poses a serious programming challenge to the development of applications. At the same time, the data exchange mode across computing power systems often depends on specific field applications, so there is a significant opportunity to optimize network data transmission at the application level. Finally, for data providers, the needs of various users and applications are often different, and fixed data access interfaces cannot adapt to diverse operation logic. SUMMARY

[0005] Therefore, in order to solve the problem of serious redundant data transmission and additional operation requests caused by the incomplete matching between the interface and the demand in the existing exchange system, the present application proposes a programmable data exchange system, which comprises:

[0006] On the data usage side, the programming language of the data usage side is expanded to allow users to write memory access subroutines in the source code;

[0007] A memory accessor is a small piece of code closely related to data access, defined as a function (sub-procedure). It runs independently of location and can be sent to a remote computing system for execution. It includes message accessors and streaming accessors. The former transmits complete message packets and exits after execution, exchanging data in a request-response model. The latter can run continuously on the remote system, continuously exchanging messages through a streaming channel.

[0008] On the data provider side, a provider service is deployed; this service is initiated by the provider runtime and can be modified by users through custom plug-ins. The data provider service program can operate with normal user privileges, without the need for intervention from the computing center administrator, and can therefore be directly deployed on existing computing systems.

[0009] On the data usage side, the application works in MPI multi-process mode. On the remote data provider side, the memory access sub-process also runs in parallel in MPI multi-process mode. Through this setting, the data transmission problem in this many-to-many situation is solved, the underlying details are shielded from the application developer, and the writing of application code is greatly simplified.

[0010] Based on the above solution, the present invention provides a programmable data exchange system. On the data-consuming side, a portion of data-intensive program code (referred to as a "memory access sub-program") is encapsulated and sent to a remote system for execution. The memory access sub-program, written by application developers, performs data preprocessing tasks such as filtering, compression, encoding, and streaming on the remote system based on application requirements. This reduces the amount of data exchanged between the two systems and the number of interactions, significantly improving overall application performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a code diagram of the present invention taking a C language program pseudo code including a memory access subroutine as an example;

[0012] Figure 2 This is a schematic diagram of the code for filtering even columns of a file and transposing and returning them. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0014] It should be noted that only parts related to the present application are shown in the drawings for the convenience of description. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0015] It should be understood that the terms “system”, “apparatus”, “unit” and / or “module” used in the present application are a method for distinguishing different components, elements, parts, sections or assemblies of different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0016] As indicated in the present application and claims, unless the context clearly indicates otherwise, the words “one”, “an”, “a” and / or “the” do not specify a singular number, but can include a plural number. Generally, the terms “comprise” and “include” only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements. The element defined by the phrase “comprising a” does not exclude the presence of additional identical elements in the process, method, product or device comprising the element.

[0017] The present application proposes an optional example of a programmable data exchange system, and the overall workflow of the system is as follows:

[0018] Firstly, the user should deploy the service of the providing side of the system of the present application on the computing power system where the data is located. The service is started by the providing side runtime in the present application, and is allowed to be modified by the user through custom plug-ins. For example, the present system can be designed to provide two plug-ins of POSIX and ODBC, which are used to support file system access and SQL database access on the computing power system, respectively. The user can adjust the two plug-ins through a configuration file to manage the real data resources accessed by them. For other types of data resources, the user can also create custom plug-ins through the providing side development kit in the present application. When building a custom plug-in, the user needs to first write an interface definition, then use the development kit to generate an interface declaration file and a framework code of the plug-in, subsequently write the plug-in function code based on the framework code, and finally compile and build the plug-in and place it in the providing side runtime.

[0019] Then, on the data usage side, the application extends the programming language C / C++ / Fortran on the data usage side, and the extended language allows users to write a memory access sub in the source code. The development kit on the usage side in the application can separate the source code developed using the extended language into a memory access sub program and a normal source code program at compile time, the former is compiled using a special compiler (invisible to users); the latter is compiled using a normal compiler, and then the two are fused into a single application program through a linking process. For the custom plug-in on the data providing side, the corresponding interface declaration file must be introduced at compile time. Without the declaration, the custom plug-in cannot use its functions, but other existing functions are not affected. The running and usage of the program are completely determined by the application, but its running startup depends on the runtime support on the usage side in the application.

[0020] Finally, during the working process of the application program, the data exchange process of the systems on both sides is completely taken over by the runtime on the usage side. The runtime on the usage side will be connected to the runtime on the providing side to jointly negotiate the running of the memory access sub and complete the data exchange. Usually, the memory access sub program is sent to the remote data providing side system for running, but in special cases (such as overload or resource shortage of the remote computing system), the memory access sub can also be run locally. The decision-making process is completely determined by the runtime system and is invisible to the application and the developer.

[0021] The embodiment allows developers to freely write access and operation logic for remote data and thus autonomously select data to be exchanged and encoding methods by stripping part of data access intensive code into independent functions that can be remotely run. Among them, the memory access sub function can be freely programmed to combine the service interface provided by the remote side in any computable way, which solves the problem that the fixed access interface in previous various ways cannot completely match the application requirements. Through reasonable programming and division of logic, the application developer can completely eliminate redundant data transmission and remote interaction.

[0022] In some feasible embodiments, a programming method and a programming model for the memory access sub on the data usage side are also provided. Referring to Figure 1 Taking a C language program pseudo code containing a memory access sub as an example, a message memory access sub cnet_function and a normal C language function host_function are given. The former will be compiled into a memory access sub program by the tool chain of the application and will be decided by the runtime whether to run locally or remotely; the latter is compiled like normal C language code and works on the local system, but since host_function uses cnet_function, the startup and running of the program must load the runtime support function of the application.

[0023] In the embodiment, the code has the following key constraints in writing:

[0024] 1. The global identifier of a function, type, global variable, etc. of a memory accessor must be appended with the keyword __cnet__;

[0025] 2. The local function can be appended with the keyword __host__ for distinction, but it can also be omitted by default;

[0026] 3. The identifier appended with __cnet__ can only refer to other identifiers also appended with __cnet__, and similarly, the identifier marked or implicitly marked with __host__ can only refer to other identifiers also marked with __host__; based on this, the two kinds of program codes can be statically strictly distinguished at the time of compilation;

[0027] 4. The function signature of a memory access is determined by the type of the memory access, and the parameters and return values are strictly predefined and fixed, and the above pseudo code gives the message memory access (cnet_mm); in addition to this, there is also a stream memory access, and the function signature form is as follows: void__cnet__cnet_ss(CNET_Stream*send, CNET_Stream*recv);

[0028] 5. The writing of the function body of a memory access is similar to that of a normal function, and any operation logic can be performed in the function body, but only the __cnet__ data structure (for example, cnet_Message and CNET_Message are different) and the predefined external function can be used;

[0029] 6. The local function can only use the defined memory access through the specified API (for example, CNET_message in the above code), and different memory access types need to be matched with different functions;

[0030] 7. The API has already mixed the MPI-based parallel programming model, for example, the second line of host_function MPI_COMM_WORLD and the fourth line n (specifying the order of the remote running of the memory access), which realizes the native support for high-performance parallel computing systems;

[0031] In addition, the embodiment also gives the related code declaration.

[0032] Table 1 Key memory access API declaration and function description

[0033]

[0034] Table 2 Key runtime API declaration and function description

[0035]

[0036] The application also provides a code example. Assuming that a data providing system "remote" has a matrix file " / matrix" stored by rows, the code for screening even columns of the file and returning the transposed result is as shown in the following table Figure 2 Compared with direct transmission of the file, the application saves about half of the data transmission amount. Compared with the RPC service of the matrix in continuous areas, the application can at least reduce m / 2 times of remote interaction. Meanwhile, the storage sub can read the file in parallel at the remote end (the parallelism is 10 in the case), which greatly improves the data acquisition performance.

[0037] The above is a specific description of the preferred embodiment of the application, but the application is not limited to the described embodiment. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the application.

Claims

1. A programmable data exchange system, characterized in that: It includes the data usage side and the data provision side, where: The data usage side allows users to write memory access sub-programs in source code; The memory access sub-program code is a program code that is IO-intensive and is remotely executed during the secure runtime of the data provider side; The said provider side service allows users to modify it through custom plug-ins; The data exchange process of the programmable data exchange system is completely taken over by the data user runtime, which connects to the provider runtime to jointly negotiate the operation of the memory access sub-module and complete the data exchange.

2. A programmable data exchange system according to claim 1, characterized in that: Also includes: Write interface definitions and generate interface declaration files and plugin framework code; Writing plug-in function code based on the framework code of the plug-in to obtain a custom plug-in; When the data providing side is running, the custom plug-in is placed.

3. A programmable data exchange system according to claim 1, characterized in that: The memory access sub-module includes a message memory access sub-module and a stream memory access sub-module, wherein: The message access sub-module transmits complete message packets, exchanges data in a request-response mode, and exits after running; The streaming memory access sub-module continuously runs at the remote end and continuously exchanges message packets through the streaming channel.

4. A programmable data exchange system according to claim 1, characterized in that: Also includes: In response to a system decision, the accessor runs locally.

5. A programmable data exchange system according to claim 1, characterized in that: Also includes: On the data usage side, the application program works in an MPI multi-process manner; On the data providing side, the memory access sub-processes run in parallel in an MPI multi-process manner.

6. A programmable data exchange system according to claim 1, characterized in that: Also includes: Adds a preset keyword before global identifiers.

7. A programmable data exchange system according to claim 6, characterized in that: Also includes: During coding, references are restricted to identical identifiers.

Images