Data processing and interaction system and method based on algorithm modular packaging

Abstract

The invention discloses a data processing and interaction system and method based on algorithm modular packaging, and relates to the field of machine learning and artificial intelligence, and the system comprises a data collection module which collects data input by a user, accesses a data source required by a client, and establishes an HTTP data set, a JSON data set and a script data set; the data processing module verifies, cleans, converts and standardizes the collected data, and evaluates data assets; the algorithm interface module is used for packaging an existing algorithm, exposing an algorithm interface, publishing a URL (Uniform Resource Locator) of the packaged algorithm, and performing data interaction by utilizing POST and GET methods; and the operation display module is used for designing a visual interface and is used for user operation data input, algorithm selection, algorithm operation and result viewing. According to the data processing and interaction system and method based on algorithm modular packaging, efficient interaction and dynamic scheduling with multiple algorithms can be achieved, and therefore the accuracy, speed and adaptability of data processing are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning and artificial intelligence technology, and in particular to a data processing and interaction system and method based on algorithm modular encapsulation. Background Art

[0002] The interaction between data and algorithms is crucial for extracting useful information and making data-driven decisions. The interaction process between data and algorithms involves data input, processing, and output results. After data is collected and preprocessed, it is input into the algorithm. The algorithm processes the data according to preset rules or knowledge obtained through learning to generate predictions, classifications, clustering, or other forms of analysis results. These results can be used for decision support, process optimization, etc. Traditional algorithm implementations often face the problem of exposing implementation details when processing complex data, which makes the system difficult to maintain and expand. In addition, directly exposing the algorithm implementation may also sacrifice the user interaction experience because users are exposed to complex technical details rather than simplified interface operations. With the continuous growth of data volume and the increasing complexity of algorithms, it is becoming increasingly important to develop data processing methods that can interact efficiently with algorithms. These methods must not only be able to handle large-scale and complex data sets, but also need to consider the interpretability and transparency of the algorithm to meet the needs of different application fields. Therefore, researching and developing new data processing technologies is an important topic in the current field of data science and machine learning. Summary of the invention

[0003] The purpose of the present invention is to provide a data processing and interaction system and method based on algorithm modular encapsulation to solve the problems mentioned in the above background technology.

[0004] To achieve the above objectives, the present invention provides a data processing and interaction system based on algorithm modular encapsulation, comprising:

[0005] The data collection module collects data entered by users and can also access the data sources required by customers to establish HTTP data sets, JSON data sets, and script data sets;

[0006] The data processing module verifies, cleans, converts and standardizes the collected data, and conducts data asset evaluation to assess the value of the stored data;

[0007] The algorithm interface module is used to encapsulate the existing algorithm and only expose the algorithm interface to ensure the privacy of the algorithm call; publish the encapsulated algorithm to a URL in accordance with the HTTP protocol and use the POST and GET methods for data interaction;

[0008] Run the display module and design a visual interface to facilitate users to input data, select algorithms, run algorithms, view results, etc., so that users can view the algorithm running status and make modifications.

[0009] Preferably, the algorithm interface module encapsulates the algorithm through a modular algorithm encapsulation method, which hides the algorithm implementation details behind an interface and only exposes the calling method and necessary parameters. Specifically for algorithm encapsulation and providing methods, the algorithm encapsulation is completed in two parts: one is the methods module, which contains the specific functional implementation of all algorithms; the other is the interface (API) module, which references the specific functions of the algorithm and publishes it to the specified URL; the algorithm interface development should comply with the RESTful specification, and the communication protocol should use HTTP;

[0010] Preferably, through a visual interface, the user enters the data required for the corresponding algorithm in a specific data box, integrating data interaction, algorithm operation and result display functions. After the user enters data in this visual interface, the data is automatically matched with the algorithm, and the algorithm will obtain the corresponding algorithm parameter values ​​from the input data.

[0011] Preferably, the visualization interface includes an input module, an algorithm module and an output display module, and is displayed in the form of graphic blocks. The user drags a considerable number of input modules, required algorithm modules, and a considerable number of output display modules, and connects them. The input module can be bound to a data source generated by a corresponding data set, and connection with the algorithm module indicates inputting data into the algorithm, the data is automatically matched with the algorithm, and the algorithm parameters in the algorithm are assigned values. The output display module is used to display the running results and running status codes.

[0012] Preferably, the HTTP data set includes a URL accessed by user input data, and a portion of the input data retained in conjunction with the data processing command;

[0013] JSON datasets allow users to input JSON data into a visual interface to create datasets, and also allow users to write instructions to modify datasets;

[0014] The script dataset supports users to write scripts to obtain data in a richer form. Users can freely choose the method of obtaining data. After obtaining the data, users can write instructions to modify it.

[0015] Preferably, the data processing module processes the collected data specifically including: checking the data format, scope and legality, removing duplicate values ​​and filling in valid data, converting the data into the format required by the corresponding algorithm and standardizing it, and performing data asset assessment on data quality, integrity and availability.

[0016] A data processing and interaction method based on algorithm modular encapsulation, comprising the following steps:

[0017] S1. User login and data collection. The user logs in to verify his or her identity in the visual interface of the data processing and interaction system, performs identity authentication and authorization, and the user directly inputs data or creates a data set to input data, and collects the data input by the user end;

[0018] S2. Data processing: data processing of collected data, including data verification and cleaning, data conversion and standardization, and data asset evaluation;

[0019] S3, algorithm encapsulation, modularize the algorithm, expose the interface, hide the specific function implementation for system call;

[0020] S4, data transmission and algorithm processing. The user operates the input module, algorithm module and output display module in the visual interface, and transmits the processed data to the algorithm for processing. After receiving the data, the algorithm performs corresponding operations. If there is an error in the input data, a corresponding value will be generated. Each value represents an error type. After the processing is completed, the algorithm will generate the corresponding result data;

[0021] S5. Output the operation results. After the algorithm processing is completed, the result data is returned to the visualization interface, and the calculated results are output in JSON format to visualize the calculation process.

[0022] Preferably, the interface information exposed by the algorithm encapsulation in S5 includes API endpoints, input parameters, error handling, version management, documentation, and examples.

[0023] Therefore, the present invention adopts the above-mentioned data processing and interaction system and method based on modular encapsulation of algorithms, which can allow the system to achieve efficient data processing and user interaction experience through interactive operations during the data processing process; and through efficient interaction with the modularly encapsulated algorithm, more refined and faster data processing can be achieved; it can achieve efficient interaction and dynamic scheduling with multiple algorithms, thereby improving the accuracy, speed and adaptability of data processing.

[0024] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of functional modules of a data processing and interaction system based on algorithm modular encapsulation according to the present invention;

[0026] Figure 2 A schematic diagram of the structure of a data processing and interaction system based on algorithm modular encapsulation according to the present invention;

[0027] Figure 3 It is a flow chart of the data processing and interaction method of the present invention;

[0028] Figure 4 This is a schematic diagram of the principle of modular encapsulation of algorithms in the present invention. DETAILED DESCRIPTION

[0029] Example

[0030] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] like Figure 1-Figure 2 As shown, a data processing and interaction system based on algorithm modular encapsulation includes:

[0032] The data collection module collects data entered by users and can also access the data sources required by customers to establish HTTP data sets, JSON data sets, and script data sets;

[0033] The data processing module verifies, cleans, converts and standardizes the collected data, and conducts data asset evaluation to assess the value of the stored data;

[0034] The algorithm interface module is used to encapsulate the existing algorithm and expose the algorithm interface. It publishes the encapsulated algorithm to a URL in accordance with the HTTP protocol and uses the POST and GET methods for data interaction.

[0035] Run the display module and design a visual interface to facilitate users to input data, select algorithms, run algorithms, view results, etc., so that users can view the algorithm running status and make modifications.

[0036] like Figure 3-Figure 4 As shown, a data processing and interaction method based on algorithm modular encapsulation, applying the above data processing and interaction system, includes the following steps:

[0037] The user first logs in and verifies his or her identity in the visual interface of the data algorithm interaction system;

[0038] Use tokens to ensure the privacy and security of data and algorithm interactions. Perform identity authentication and authorization. Users or systems need to perform identity authentication when they interact for the first time.

[0039] After the system verifies the user's identity, it issues a unique access token. The token usually contains the user's identification information, permission scope, validity period, etc. Data encryption is performed before data transmission. Symmetric or asymmetric encryption technology is used to ensure that only users with the correct key can decrypt the data;

[0040] Users can enter or select data through text boxes, file uploads, selection boxes, etc. The data can be text, numbers, images, audio, or other forms of information. For example, a form can contain a text input box for entering text data, a number input box for entering numerical data, a file upload button for uploading images or audio files, a selection box for selecting predefined options, etc.

[0041] Users can also create data sets in the system to input large amounts of data. They can enter the URL of the data to create an HTTP data set, which the system can access on its own, then create the corresponding data set and save it to the database.

[0042] Users can also create and modify data sets by writing scripts, and can create JSON data sets and script data sets.

[0043] Data processing of collected data includes data verification and cleaning, data conversion and standardization, data assets and data evaluation.

[0044] Data validation: Before data is passed into the algorithm, it is first validated. This includes checking the data format, range, and legality. For example, whether the text input is empty, whether the numerical input is within a reasonable range, and whether the uploaded file is in an allowed format (such as JPEG images or MP3 audio). If the data does not meet the requirements, the user will be promptly fed back with specific error information, requiring the user to re-enter or select the correct data.

[0045] Data cleaning: After verification, data cleaning is performed. This includes removing duplicate values, processing missing data, correcting format errors, and other operations. For example, for text data, remove extra spaces and special characters; for numerical data, fill or remove missing values; for image data, adjust the size and format to ensure that it meets the input requirements of the algorithm. The purpose of data cleaning is to ensure that the data passed into the algorithm is accurate and of high quality.

[0046] Data conversion: converting the data input by the user into the specific format required by the algorithm. For example, converting text data into vector representation, converting numerical data into standard units, converting image data into grayscale images or matrices of specific sizes, etc. During the conversion process, operations such as data type conversion and encoding conversion may be required to ensure that the data format meets the input specifications of the algorithm.

[0047] Data normalization: Normalize the data to ensure that different input data have the same scale and distribution. For example, normalize numerical data to adjust its value range to between 0 and 1; normalize pixel values ​​of image data to adjust the pixel value range to between 0 and 255. Normalization helps improve the stability and accuracy of the algorithm.

[0048] Data assets and asset assessment: Data assets refer to all data owned by an organization or individual, which are valuable for business operations and decision-making. Including data type, data source, and data value; here we need to assess data assets to understand their quality, integrity, availability, and security. In the above data processing process, we need to assess data quality.

[0049] Perform algorithm encapsulation;

[0050] The modular encapsulation algorithm is divided into two modules. One is the Methods module, which contains the specific functions of the algorithm. A run function is written in the algorithm function program, and the names of the algorithm parameters and input parameters are defined to accurately obtain the corresponding values ​​from the custom_params (algorithm parameters) and custom_input (input parameters) dictionaries. The other module is the Api module, which contains the design of the algorithm interface. The POST function is written in the interface program to process the incoming data type and use the add_resource method to generate the URL of the corresponding algorithm.

[0051] This encapsulation method only exposes the interface that reflects the algorithm information and hides the specific implementation process of the algorithm.

[0052] Algorithm interface information to be exposed in the algorithm encapsulation module:

[0053] API endpoint: uses the POST method to accept requests and send responses;

[0054] Input parameters: declare the input data format, algorithm parameters and input parameter names;

[0055] Error handling: define possible error codes and error messages, as well as suggested methods for handling errors. This helps the caller better understand and respond to possible problems;

[0056] Version management: If the service may be updated or evolved, it is recommended to provide a version control mechanism to ensure compatibility and smooth transition between new and old versions.

[0057] Documentation and Examples: Provide clear interface documentation that describes the purpose, parameters, expected responses, and examples of each endpoint to help developers quickly understand and use the service.

[0058] The algorithm program of this embodiment is obtained from the custom_params (algorithm parameters) and custom_input (input parameters) dictionaries. Use numpy to convert the data into a suitable format (such as a NumPy array). The converted data is passed as a parameter to the fit method of the algorithm for training. The trained model is serialized and returned using the dill module.

[0059] Next, the algorithm interface module passes the data to the algorithm:

[0060] Data delivery: Once the data has been validated, cleaned, transformed, and standardized, it can be passed to the algorithm for processing. This may involve passing the data as a parameter to the algorithm's function or API, or storing the data in a specific data structure for the algorithm to access and process. For example, the standardized data is packaged into JSON format and passed to the algorithm API on the server side through an HTTP request.

[0061] Algorithm processing: After receiving the data, the algorithm performs corresponding processing and calculation. The algorithm processing process may involve complex calculations, model predictions, data analysis and other operations. After the processing is completed, the algorithm will generate the corresponding result data.

[0062] Result return: After the algorithm processing is completed, the result data is returned to the UI interface. The result data can be in various forms such as text, numerical values, charts, images, etc. Users can directly view the processing results in the interface and get timely feedback. For example, for a text analysis algorithm, the result may be a sentiment analysis score; for an image processing algorithm, the result may be a processed image or recognition result.

[0063] The data processing and interaction system and method of this embodiment are applicable to multiple fields, including but not limited to business analysis, medical diagnosis, industrial production optimization, etc., and provide users and enterprises with powerful data processing and decision support capabilities.

[0064] Therefore, the present invention adopts the above-mentioned data processing and interaction system and method based on algorithm modular encapsulation to ensure that users can obtain timely and accurate feedback and results during the interaction between the UI interface and the algorithm. The entire process, from data reception to result return, covers important links such as data verification and cleaning, data conversion and standardization, thereby ensuring the accuracy and effectiveness of data processing and improving user experience.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A data processing and interaction system based on algorithm modular encapsulation, characterized in that: include: Data collection module, which collects data input by users, accesses data sources required by customers, and establishes HTTP data sets, JSON data sets, and script data sets; The data processing module verifies, cleans, converts and standardizes the collected data, and conducts data asset evaluation; The algorithm interface module is used to encapsulate the existing algorithm and expose the algorithm interface, publish the encapsulated algorithm to the URL, and use the POST and GET methods for data interaction; Run the display module and design a visual interface for users to input data, select algorithms, run algorithms, and view results.

2. A data processing and interaction system based on algorithm modular encapsulation according to claim 1, characterized in that: The algorithm interface module encapsulates the algorithm through a modular algorithm encapsulation method, which hides the algorithm implementation details behind an interface and exposes the calling method and parameters.

3. A data processing and interaction system based on algorithm modular encapsulation according to claim 1, characterized in that: Through the visual interface, the user enters the data required by the corresponding algorithm in a specific data frame. After the user enters the data in the visual interface, the data is automatically matched with the algorithm, and the algorithm will obtain the corresponding algorithm parameter value from the input data.

4. The data processing and interaction system based on algorithm modular encapsulation according to claim 1, characterized in that: The visualization interface includes input modules, algorithm modules and output display modules, and is displayed in the form of graphic blocks. Users drag a number of input modules, required algorithm modules, and a number of output display modules and connect them. The input module is bound to the data source generated by the corresponding data set, and the connection with the algorithm module indicates that data is input to the algorithm. The data and the algorithm are automatically matched, and the algorithm parameters in the algorithm are assigned. The output display module is used to display the running results and running status codes.

5. The data processing and interaction system based on algorithm modular encapsulation according to claim 1, characterized in that: The HTTP data set contains the URL accessed by the user input data, and the input data portion retained in conjunction with the data processing command; JSON datasets allow users to input JSON data into a visual interface to create datasets, and also allow users to write instructions to modify datasets; The script dataset supports users to write scripts to obtain data and freely choose the method of obtaining data. After obtaining the data, users can write instructions to modify it.

6. The data processing and interaction system based on algorithm modular encapsulation according to claim 1, characterized in that: The data processing module processes the collected data in the following ways: checking the data format, scope and legality, removing duplicate values ​​and filling in valid data, converting the data into the format required by the corresponding algorithm and standardizing it, as well as conducting data asset assessment on data quality, integrity and availability.

7. A data processing and interaction method based on algorithm modular encapsulation, characterized in that: The following steps are involved: S1. User login and data collection. The user logs in to verify his or her identity in the visual interface of the data processing and interaction system, performs identity authentication and authorization, and the user directly inputs data or creates a data set to input data, and collects the data input by the user end; S2. Data processing: data processing of collected data, including data verification and cleaning, data conversion and standardization, and data asset evaluation; S3, algorithm encapsulation, modularize the algorithm, expose the interface, hide the specific function implementation for system call; S4, data transmission and algorithm processing, the data is verified, cleaned, converted and standardized, and then passed to the algorithm for processing. After the algorithm receives the data, it performs corresponding processing and calculation. After the processing is completed, the algorithm will generate the corresponding result data; S5. Output the operation results. After the algorithm processing is completed, the result data is returned to the visualization interface, and the calculated results are output in JSON format to visualize the calculation process.

8. The data processing and interaction method based on algorithm modular encapsulation according to claim 7, characterized in that: The interface information exposed by the algorithm encapsulation in S5 includes API endpoints, input parameters, error handling, version management, documentation, and examples.

Images