A general algorithm verification platform and an algorithm registration and calling method thereof

Abstract

This invention discloses a general algorithm verification platform and its algorithm registration and invocation method. The platform provides an open-source algorithm plugin framework template, which includes multiple languages. Users integrate algorithms using this framework and fill in the processing results according to the network interface protocol requirements for algorithm-platform interaction. User algorithms do not need to consider network communication. The interface between the algorithm and the platform uses TCP / IP communication. The uploaded algorithm acts as a TCP client, actively initiated by the platform, and receives connection information from the TCP server, as well as initialization parameters. The algorithm continuously attempts to connect with the platform. When a connection is successful, it automatically sends process ID information, waits for platform data input, the platform transmits data to the algorithm, and the algorithm feeds back the processing results to the platform. This invention facilitates the verification and integration of different algorithms, thereby minimizing user development costs.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing technology, specifically a general algorithm verification platform and its algorithm registration and invocation method. Background Technology

[0002] Existing general-purpose distributed intelligent processing and analysis platforms for electromagnetic signals (such as patent CN113157252A) can realize distributed computing functions in a distributed computing cluster. This distributed intelligent processing and analysis platform consists of a scheduling node, several regular computing nodes, several advanced computing nodes, and several data access nodes. Each node deploys a node management module to monitor hardware status, software running status, and start / stop software modules. The scheduling node deploys a scheduling module, serving as the system's central hub, responsible for scheduling and resource coordination. The computing nodes are used to adapt to parallel processing tasks. The UI front-end module receives user task instructions and sends them to the back-end service module via a communication protocol. After successful resource monitoring and allocation by the node management module, the relevant modules (data access module, preprocessing computing module, deep learning computing module) start the task. Each module interacts with the data management module to achieve data storage and management. After the task is completed, the results are fed back via a communication protocol. The back-end service module is responsible for task scheduling, the node management module is responsible for resource reclamation, and the UI front-end module is responsible for result display.

[0003] The software framework of the aforementioned platform includes an Nginx front-end module, a scheduling module, an agent assistant software module, a data management module, several data access modules, several regular computing software modules, and several advanced computing software modules. It can run on a single computer or on multiple computers according to roles. For example, the data access software module runs on the acquisition server, and there are two types of computing nodes: regular computing software modules run on regular computing nodes, and advanced computing software modules run on advanced computing nodes.

[0004] In distributed computing and multi-task concurrency scenarios, multiple clients will simultaneously send task requests to the scheduling software, and each user may request multiple tasks. Furthermore, each task will have parallel computing processes, even distributed across different computing nodes. The essence of this problem is that the scheduling module must be able to accept multiple task requests and, based on the current state of computing resources (including both conventional and advanced computing software modules), rationally allocate tasks to the appropriate computing resources, while simultaneously monitoring and managing resource operation.

[0005] The algorithm of the platform in patent CN113157252A communicates through the conventional computing software module of the conventional computing node, and the user logic is also written in the conventional computing software module. However, it is difficult for users to verify and integrate different algorithms, resulting in high development costs for users. Summary of the Invention

[0006] The purpose of this invention is to provide a general algorithm verification platform and its algorithm registration and invocation method, which can facilitate users to verify and integrate different algorithms and minimize users' development costs.

[0007] The technical solution to achieve the purpose of this invention is: a general algorithm verification platform, which provides an open source algorithm plugin framework template. The algorithm plugin framework includes multiple languages. Users integrate algorithms with the algorithm plugin framework and fill in the processing results according to the network interface protocol requirements for the interaction between the algorithm and the platform. The user algorithm does not consider network communication-related content.

[0008] The interface between the algorithm and the platform uses TCP / IP communication. The uploaded algorithm acts as a TCP client, which is actively started by the platform and receives the connection information of the TCP server, as well as the initialization parameter information. The algorithm and the platform continuously attempt to connect. When the connection is successful, the algorithm automatically sends information such as the process ID and waits for data input from the platform. The platform then transmits the data to the algorithm, and the algorithm sends the processing results back to the platform.

[0009] An algorithm registration and invocation method based on the aforementioned general algorithm verification platform, wherein the method performs algorithm registration and synchronization based on a node management module running on a node and a scheduling module on a scheduling node, wherein the scheduling module is the main running process of the program, and one node management module runs on each node; and the algorithm is invoked based on a conventional computing software module on a conventional computing node.

[0010] By logging into the system and registering the algorithm, the system automatically synchronizes the algorithm to all computing nodes. Users can then verify the algorithm by writing and calling the processing logic.

[0011] Compared with the prior art, the significant advantages of this invention are: (1) By designing an algorithm registration and calling method, a general algorithm verification platform is realized, which does not rely on any open source components and is independently controllable; (2) The algorithm plugin framework has multiple languages ​​such as Python and C++. Users can integrate algorithms with the help of the algorithm plugin framework and fill in the processing results according to the network interface protocol requirements for the interaction between the algorithm and the platform. The user algorithm does not need to consider network communication related content and can run on multiple operating systems; (3) The messages of the interaction between the platform and the algorithm can be automatically adapted according to the content without adjusting the message format, which facilitates the user to verify and integrate different algorithms and minimizes the user's development cost. Attached Figure Description

[0012] Figure 1 This is a diagram of the background system hardware composition.

[0013] Figure 2 This is the background system software architecture diagram.

[0014] Figure 3 This is an example diagram of a usage scenario.

[0015] Figure 4 This is a flowchart of the platform algorithm call process. Detailed Implementation

[0016] Combination Figures 1-2 The present invention provides a general algorithm verification platform, which provides an open-source algorithm plugin framework template. The algorithm plugin framework includes multiple languages. Users integrate algorithms using the algorithm plugin framework and fill in the processing results according to the network interface protocol requirements for algorithm-platform interaction. The user algorithm does not consider network communication related content.

[0017] The interface between the algorithm and the platform uses TCP / IP communication. The uploaded algorithm acts as a TCP client, which is actively started by the platform and receives the connection information of the TCP server, as well as the initialization parameter information. The algorithm and the platform continuously attempt to connect. When the connection is successful, the algorithm automatically sends information such as the process ID and waits for data input from the platform. The platform then transmits the data to the algorithm, and the algorithm sends the processing results back to the platform.

[0018] In one specific implementation, the algorithm interacts with the platform using a unified format of JSON with a bin file attachment, and the input and output parameters filled in during the upload and registration process are parsed and processed.

[0019] In one specific implementation, the user uploads and registers the algorithm to the system. When uploading, the user fills in the algorithm name, initialization parameters, and whether the algorithm is used. The system automatically synchronizes the algorithm to each computing node. Each computing node selects whether to start the algorithm based on the usage flag and automatically maintains the algorithm call mapping table.

[0020] In one specific implementation, the algorithm plugin framework uses languages ​​including Python and C++.

[0021] Combination Figures 3-4 This invention provides an algorithm registration and invocation method based on the aforementioned general algorithm verification platform. This method performs algorithm registration and synchronization based on a node management module running on a node and a scheduling module on a scheduling node. The scheduling module is the main running process of the program, and one node management module runs on each node. Algorithm invocation is performed based on a conventional computing software module on a conventional computing node.

[0022] By logging into the system and registering the algorithm, the system automatically synchronizes the algorithm to all computing nodes. Users can then verify the algorithm by writing and calling the processing logic.

[0023] In one specific implementation, the method is used in the following two scenarios: one is a single interface that accommodates multiple algorithms, allowing the algorithms to be selected for verification testing; the other is different interfaces that, based on different algorithms, require writing call processing logic to implement specific software functions.

[0024] In one specific implementation, the method includes three processes: algorithm uploading, algorithm registration and initiation, and algorithm invocation.

[0025] In one specific implementation, the algorithm upload specifically involves:

[0026] (1.1) Users upload algorithm plugins to the platform's data management module through the platform's algorithm management interface. When uploading, click the upload button in the algorithm management interface and fill in the algorithm name, algorithm category, initialization parameters, algorithm startup relative path and whether to use this information in the pop-up window.

[0027] (1.2) After receiving the algorithm plugin and description information uploaded by the front end, the platform's data management module generates the corresponding algorithm ID according to the algorithm category and stores this information in the database according to the absolute startup path set by the platform.

[0028] (1.3) The notification scheduling module transmits the algorithm plugin to the node management module on each computing node through network communication. The node management module on the node stores the algorithm module in the specified path, so that the platform can automatically synchronize the algorithm to each computing node.

[0029] In one specific implementation, the algorithm registration and startup process is as follows:

[0030] (2.1) The user creates a new data processing and analysis task and selects the algorithm to be called in the process;

[0031] (2.2) When the conventional computing software receives the task to be processed, it parses the task flow parameters, finds the algorithm plugin of this node according to the algorithm ID, and starts the algorithm plugin.

[0032] (2.3) After the algorithm plugin is started, the algorithm, as a TCP client, establishes a connection with the conventional computing software module according to the startup parameters of the conventional computing software module;

[0033] (2.4) Each conventional computing software module maintains information such as network connection ID, algorithm name, and process ID with the algorithm plugin in a mapping table, and the algorithm is registered in the platform.

[0034] In one specific implementation, the algorithm call is as follows:

[0035] (3.1) When writing business logic that requires algorithm calls, users call the platform's encapsulated API, pass in the algorithm name, function name and input structure to call the algorithm, and implement synchronous messages, asynchronous messages and delayed messages by setting them;

[0036] (3.2) The input structure is in the format specified by the platform, including the overall structured information description, JSON description length, data body length, JSON, and data body; attribute and parameter information are written into JSON, and the data body stores the data to be processed, which is unstructured or semi-structured data. The platform performs serialization operation on the input data.

[0037] (3.3) The platform API notifies the regular computing software to transmit the data to be processed to the algorithm plugin via TCP / IP;

[0038] (3.4) The conventional computing software monitors the result return information of the algorithm plugin. When the algorithm plugin returns the structure, it performs deserialization operation and stores it in the processing result queue of the algorithm. The system then uses the platform API to notify the user's business logic algorithm call location.

[0039] (3.5) Users obtain data processing results through the platform API based on the received notification information.

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] Example

[0042] The hardware composition diagram of the algorithm registration and invocation method of the general algorithm verification platform described in this invention is as follows: Figure 1 As shown, the structure of the distributed computing software framework is as follows: Figure 2 As shown.

[0043] The algorithm registration and invocation method of the general algorithm verification platform described in this invention relies on the node management module running on the node and the scheduling module on the scheduling node for algorithm registration and synchronization. The scheduling module is the main process running the program, and one node management module runs on each node; the algorithm invocation relies on the conventional computing software module on the conventional computing node.

[0044] Based on the platform described in patent CN113157252A, this invention designs and develops an algorithm registration and invocation method to improve the development cost of a general algorithm verification platform.

[0045] This invention implements a general algorithm verification platform by designing an algorithm registration and invocation method. A general algorithm refers to an algorithm that, without considering its internal implementation details, can be encapsulated and invoked to obtain the processing result from the input data.

[0046] The present invention discloses a general algorithm verification platform algorithm registration and invocation method, which relies on the node management module on the platform and each computing node to realize algorithm registration and node synchronization, and relies on conventional computing software running on conventional computing nodes to realize algorithm invocation and interaction functions.

[0047] The technical solution of this invention is as follows: The general algorithm verification platform algorithm registration and calling method described in this invention allows users to upload and register algorithms to the system. When uploading, users fill in information such as algorithm name, initialization parameters, and whether the algorithm is used. The system automatically synchronizes the algorithm to each computing node. Each computing node selects whether to start the algorithm based on the usage flag and automatically maintains the algorithm calling mapping table.

[0048] The platform provides open-source algorithm plugin framework templates, which are available in multiple languages ​​such as Python and C++. Users can integrate algorithms using this framework and fill in the processing results according to the network interface protocol requirements for the interaction between the algorithm and the platform. Users do not need to consider network communication-related aspects of their algorithms.

[0049] The interface between the algorithm and the platform uses TCP / IP communication. The uploaded algorithm acts as a TCP client, which is actively started by the platform and receives the connection information of the TCP server, as well as the initialization parameter information. The algorithm and the platform continuously attempt to connect. When the connection is successful, the algorithm automatically sends information such as the process ID and waits for data input from the platform. The platform then transmits the data to the algorithm, and the algorithm sends the processing results back to the platform.

[0050] The algorithm interacts with the platform using a unified format of JSON with a bin file attachment. The communication is based on the input / output parameters provided during registration and upload. The system relies on a node management module running on dependent nodes and a scheduling module running on scheduling nodes. The scheduling module is the main process running on the program, while one node management module runs on each node.

[0051] The algorithm registration and invocation method based on the general algorithm verification platform of the present invention includes three processing steps: algorithm uploading, algorithm registration initiation, and algorithm invocation.

[0052] 1. Algorithm Upload:

[0053] Users upload algorithm plugins to the platform's data management module through the algorithm management interface. During upload, users click the upload button in the interface and fill in information such as algorithm name, algorithm category, initialization parameters, relative startup path, and whether the algorithm is being used in a pop-up window. Upon receiving the uploaded algorithm plugin and description, the platform's data management module generates a corresponding algorithm ID based on the algorithm category and stores it in the database according to the absolute startup path set by the platform. The notification and scheduling module transmits the algorithm plugin to the node management module on each computing node via network communication. The node management module then stores the algorithm module in a specified path, enabling the platform to automatically synchronize algorithms to all computing nodes.

[0054] 2. Algorithm registration and startup:

[0055] A user creates a new data processing and analysis task and selects the algorithm to be used in the workflow. The platform, based on the algorithm ID, launches the regular computing software module. According to the startup parameters of the regular computing software module, the algorithm, acting as a TCP client, establishes a connection with the module. Each regular computing software module maintains a mapping table (containing information such as TCP connection ID, algorithm name, and process ID), and the algorithm is registered in the platform. The specific process is as follows:

[0056] 1) The user creates a new data processing and analysis task and selects the algorithm to be called in the workflow;

[0057] 2) When the conventional computing software receives the task to be processed, it parses the task flow parameters, finds the algorithm plugin for this node based on the algorithm ID, and starts it.

[0058] 3) After the algorithm plugin starts, it acts as a TCP client to establish a connection with the regular computing software module;

[0059] 4) Conventional computing software maintains information such as network connection ID, algorithm name, and process ID of the algorithm plugin in a mapping table.

[0060] 3. Algorithm call:

[0061] When user logic invokes the algorithm, it only needs to call the platform's API, passing in the algorithm name, function name, and input structure. The platform, based on the function name, finds the corresponding ID, then locates the bound algorithm IP and port number, converts the input structure into a binary stream, and sends it to the algorithm. The algorithm returns a binary stream of results and converts it according to the output structure. The input and output structures have the same format, both consisting of JSON objects and byte arrays. The JSON object in the input structure carries input parameters, attributes, and other information, while the byte array carries the data to be processed. The JSON object in the output structure carries the return result information, while the byte array in the input structure carries the binary data to be returned. The platform deserializes the algorithm's return result and provides the processing result to the user's business logic. The specific process is as follows:

[0062] 1) When users need to call algorithms in their business logic, they can call the platform's encapsulated API, pass in the name of the algorithm, the name of the function, and the input structure to call the algorithm. Synchronous messages, asynchronous messages, and delayed messages can be configured.

[0063] 2) The input structure is in the format specified by the platform, including a structured overall description of the information, the length of the JSON description, the length of the data body, the JSON, and the data body; general attribute and parameter information is written into the JSON, and the data body stores the data to be processed, which is generally unstructured or semi-structured data. The platform performs serialization operations on the input data.

[0064] 3) The platform API notifies the regular computing software to transmit the data to be processed to the algorithm plugin via TCP / IP;

[0065] 4) Conventional computing software monitors the result return information of the algorithm plugin in one thread. When it receives the return structure of the algorithm plugin, it performs deserialization operation and stores it in the processing result queue of the algorithm. Then, it notifies the user's business logic algorithm call point through the platform API.

[0066] 5) Users can obtain data processing results through the platform API based on the received notification information.

[0067] The algorithm registration and invocation method of the general algorithm verification platform described in this invention is applicable to two scenarios: one is the same interface that can accommodate multiple algorithms, and the algorithms can be selected for verification testing; the other is different interfaces, and specific software functions are implemented by writing invocation processing logic according to different algorithms.

[0068] By logging into the system and registering the algorithm, the system automatically synchronizes the algorithm to all computing nodes. Users can verify the algorithm by writing the call processing logic. Users do not need to worry about details such as network communication, which is simpler than the RPC method.

[0069] The universal algorithm verification platform and its algorithm registration and invocation method described in this invention are applicable to two scenarios: one is a single interface that accommodates multiple algorithms, allowing users to select algorithms for verification testing; the other is different interfaces where specific software functions are implemented by writing invocation processing logic based on different algorithms. By logging into the system and registering the algorithm, the system automatically synchronizes the algorithm to all computing nodes. Users verify the algorithm by writing invocation processing logic. Users do not need to concern themselves with details such as network communication, making it simpler than the RPC method.

Claims

1. A general algorithm verification platform, characterized in that, The platform provides open-source algorithm plugin framework templates, which include multiple languages. Users integrate algorithms using these frameworks and fill in the processing results according to the network interface protocol requirements for algorithm-platform interaction. User algorithms do not consider network communication-related content. The interface between the algorithm and the platform uses TCP / IP communication. The uploaded algorithm acts as a TCP client, which is actively started by the platform and passes in the connection information of the TCP server and the initialization parameter information. The algorithm and the platform continuously attempt to connect. When the connection is successful, the algorithm automatically sends information such as the process ID and waits for data input from the platform. The platform then passes the data to the algorithm, and the algorithm feeds back the processing results to the platform. Algorithm registration and invocation includes three processes: algorithm upload, algorithm registration startup, and algorithm invocation. The algorithm upload specifically involves: (1.1) Users upload algorithm plugins to the platform's data management module through the platform's algorithm management interface. When uploading, click the upload button in the algorithm management interface and fill in the algorithm name, algorithm category, initialization parameters, algorithm startup relative path and whether to use this information in the pop-up window. (1.2) After receiving the algorithm plugin and description information uploaded by the front end, the platform's data management module generates the corresponding algorithm ID according to the algorithm category and stores this information in the database according to the absolute startup path set by the platform. (1.3) The notification scheduling module transmits the algorithm plugin to the node management module on each computing node through network communication. The node management module on the node stores the algorithm module in the specified path, so that the platform can automatically synchronize the algorithm to each computing node. The specific steps for registering and starting the algorithm are as follows: (2.1) The user creates a new data processing and analysis task and selects the algorithm to be called in the process; (2.2) When the conventional computing software receives the task to be processed, it parses the task flow parameters, finds the algorithm plugin of this node according to the algorithm ID, and starts the algorithm plugin; (2.3) After the algorithm plugin is started, the algorithm, as a TCP client, establishes a connection with the conventional computing software module according to the startup parameters of the conventional computing software module; (2.4) Each conventional computing software module maintains information such as network connection ID, algorithm name, and process ID with the algorithm plugin in a mapping table, and the algorithm is registered in the platform; The algorithm call is specifically as follows: (3.1) When writing business logic that requires algorithm calls, users call the platform's encapsulated API, pass in the algorithm name, function name and input structure to call the algorithm, and implement synchronous messages, asynchronous messages and delayed messages by setting them; (3.2) The input structure is in the format specified by the platform, including the overall structured information description, JSON description length, data body length, JSON, and data body; attribute and parameter information are written into JSON, and the data body stores the data to be processed, which is unstructured or semi-structured data. The platform performs serialization operation on the input data. (3.3) The platform API notifies the regular computing software to transmit the data to be processed to the algorithm plugin via TCP / IP; (3.4) The conventional computing software monitors the result return information of the algorithm plugin. When the algorithm plugin returns the structure, it performs deserialization operation and stores it in the processing result queue of the algorithm. It then notifies the user's business logic algorithm call location through the platform API. (3.5) Users obtain data processing results by using the platform API based on the notification information they receive.

2. The general algorithm verification platform according to claim 1, characterized in that, The algorithm interacts with the platform using a unified format of JSON with a bin file attachment. The algorithm parses and processes the input and output parameters entered during the upload and registration process.

3. The general algorithm verification platform according to claim 1, characterized in that, Users upload and register algorithms to the system. When uploading, they fill in the algorithm name, initialization parameters, and whether they want to use the algorithm. The system automatically synchronizes the algorithm to each computing node. Each computing node selects whether to start the algorithm based on whether it is used and automatically maintains the algorithm call mapping table.

4. The general algorithm verification platform according to claim 1, characterized in that, The algorithm plugin framework uses languages ​​including Python and C++.

5. A method for registering and invoking an algorithm based on the general algorithm verification platform according to any one of claims 1 to 4, characterized in that, This method registers and synchronizes algorithms based on a node management module running on a node and a scheduling module on a scheduling node. The scheduling module is the main process running the program, and one node management module runs on each node. Algorithm calls are performed based on conventional computing software modules on conventional computing nodes. By logging into the system and registering the algorithm, the system automatically synchronizes the algorithm to all computing nodes. Users can verify the algorithm by writing the call processing logic. This method includes three processing steps: algorithm uploading, algorithm registration and startup, and algorithm invocation. The algorithm upload specifically involves: (1.1) Users upload algorithm plugins to the platform's data management module through the platform's algorithm management interface. When uploading, click the upload button in the algorithm management interface and fill in the algorithm name, algorithm category, initialization parameters, algorithm startup relative path and whether to use this information in the pop-up window. (1.2) After receiving the algorithm plugin and description information uploaded by the front end, the platform's data management module generates the corresponding algorithm ID according to the algorithm category and stores this information in the database according to the absolute startup path set by the platform. (1.3) The notification scheduling module transmits the algorithm plugin to the node management module on each computing node through network communication. The node management module on the node stores the algorithm module in the specified path, so that the platform can automatically synchronize the algorithm to each computing node. The specific steps for registering and starting the algorithm are as follows: (2.1) The user creates a new data processing and analysis task and selects the algorithm to be called in the process; (2.2) When the conventional computing software receives the task to be processed, it parses the task flow parameters, finds the algorithm plugin of this node according to the algorithm ID, and starts the algorithm plugin; (2.3) After the algorithm plugin is started, the algorithm, as a TCP client, establishes a connection with the conventional computing software module according to the startup parameters of the conventional computing software module; (2.4) Each conventional computing software module maintains information such as network connection ID, algorithm name, and process ID with the algorithm plugin in a mapping table, and the algorithm is registered in the platform; The algorithm call is specifically as follows: (3.1) When writing business logic that requires algorithm calls, users call the platform's encapsulated API, pass in the algorithm name, function name and input structure to call the algorithm, and implement synchronous messages, asynchronous messages and delayed messages by setting them; (3.2) The input structure is in the format specified by the platform, including the overall structured information description, JSON description length, data body length, JSON, and data body; attribute and parameter information are written into JSON, and the data body stores the data to be processed, which is unstructured or semi-structured data. The platform performs serialization operation on the input data. (3.3) The platform API notifies the regular computing software to transmit the data to be processed to the algorithm plugin via TCP / IP; (3.4) The conventional computing software monitors the result return information of the algorithm plugin. When the algorithm plugin returns the structure, it performs deserialization operation and stores it in the processing result queue of the algorithm. It then notifies the user's business logic algorithm call location through the platform API. (3.5) Users obtain data processing results by using the platform API based on the notification information they receive.

6. The algorithm registration and invocation method according to claim 5, characterized in that, This method is applicable to the following two scenarios: one is a single interface that can accommodate multiple algorithms, allowing the algorithms to be selected for verification testing; the other is different interfaces that require writing call processing logic to implement specific software functions based on different algorithms.

Images