A scientific research data analysis model service platform and a service method thereof
By building a scientific research data analysis model service platform and utilizing container technology and workflow engines, the problem of insufficient computing resources for researchers has been solved, enabling efficient sharing and expansion of models and algorithms, and improving scientific research data analysis capabilities and collaboration efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- COMP NETWORK INFORMATION CENT CHINESE ACADEMY OF SCI
- Filing Date
- 2023-02-09
- Publication Date
- 2026-07-21
AI Technical Summary
When researchers conduct data analysis in their local environment, the computing resources become insufficient as computing demands increase. Models and algorithms cannot fully utilize cloud platform resources, and the lack of a model and algorithm sharing platform hinders knowledge sharing and scientific research collaboration.
We will build a scientific research data analysis model service platform, which will adopt container technology and workflow engine to provide one-stop data analysis services, integrate machine learning algorithms and statistical analysis tools, support model componentization and service-oriented architecture, and realize distributed deployment and sharing.
Enhance the ability to analyze, process, and integrate scientific research data; promote innovation in knowledge sharing and scientific research collaboration models; support large-scale scientific research data analysis; and achieve efficient sharing and expansion of models and algorithms.
Smart Images
Figure CN116302323B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of model algorithms, data analysis, machine learning, and cloud computing, and proposes a method for constructing a scientific research data analysis model service platform. Background Technology
[0002] With the explosive growth of data volume in society and industry, data analysis models and algorithms have become a topic of common concern in both academia and industry. In industry, integrating advanced algorithms and models into cloud service platforms to provide advanced technical solutions for industry applications has become a key focus for many cloud service providers and artificial intelligence research institutions. Google has developed a cloud service platform for machine learning and artificial intelligence. Compared with other large-scale machine learning systems, this platform, based on Google's TensorFlow deep learning neural network framework, offers a wider range of machine learning models and algorithms, as well as application service capabilities. Microsoft provides the Azure Machine Learning Studio cloud service platform, covering functions such as importing datasets, training and deploying models, processing data in Jupyter Notebooks, and saving models. Major cloud service providers in China have also begun to offer cloud service platforms for machine learning and artificial intelligence to industry and society. Platforms such as Alibaba Cloud's PAI platform, Tencent Cloud's TML platform, and Baidu Cloud's BML platform support mainstream frameworks such as TensorFlow, BigDL, Caffe, and Xgboost. They also have built-in deep learning algorithms such as CNN and DNN, which can be applied to image processing and natural language processing. They provide cloud services for algorithms, data, model components, and applications.
[0003] Research activities across various fields generate and utilize complex, multi-source, and heterogeneous data. Analyzing and processing this data using diverse models and algorithms is a crucial aspect of scientific research. Unlike industry applications, research typically involves researchers directly accessing model libraries within software packages for data analysis within their local environments. For example, researchers might use Python to call models and algorithms from the scikit-learn package, or use Matlab to utilize its provided functions for data analysis. This application model of algorithms and models has the following limitations:
[0004] 1. Model-based data analysis is limited by the local computing infrastructure environment of researchers. When the amount of data that researchers need to analyze and process increases, local personal computers or small computing environments cannot meet the computing needs. It is necessary to consider combining researchers' data analysis needs with infrastructures that support elastic scaling, such as cloud computing.
[0005] 2. The models and algorithms used locally by researchers are not adapted to big data processing environments and cloud platforms. This results in the models and algorithms not being able to fully utilize the advantages of underlying resources even when deployed in a cloud environment.
[0006] 3. The lack of platforms and mechanisms for sharing models and algorithms hinders the promotion of knowledge sharing and innovation in scientific research collaboration models.
[0007] Providing a one-stop application service for scientific research data analysis models is of great significance for improving the ability to analyze, process, and integrate scientific research data, and for promoting knowledge sharing and innovation in scientific research collaboration models. Therefore, building a scientific research data analysis model service platform is a pressing need to support modern scientific research data analysis and processing.
[0008] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention
[0009] To address the technical problems existing in the prior art, the present invention aims to provide a method for constructing a scientific research data analysis model service platform. This method supports different research fields in building upon accumulated data resources, combining research and application needs, and integrating a scientific research data analysis model application and integration engine, a container-based model application environment, and a workflow engine-based service environment into a comprehensive scientific research data analysis model service platform. By constructing this platform, a one-stop scientific research data analysis and processing service can be provided, thereby enhancing the scientific research data analysis and processing capabilities and application integration capabilities across different fields.
[0010] This invention improves the applicability of analytical models to scientific research fields and enhances the sharing of knowledge, technologies, and methods in scientific research by modularizing and servicing models and algorithms and publishing and sharing them on a cloud platform. The provided scientific data analysis model library supports data mining and machine learning algorithms such as basic statistics, random sampling, classification, regression, clustering, support vector machines, Naive Bayes, neural networks, and deep learning, and supports effective expansion of the model library. Through cloud services, scientific research users can easily publish and obtain models and algorithms for scientific data analysis.
[0011] This invention utilizes container technology to construct an integrated environment for scientific research data analysis model applications. It containerizes and encapsulates provided machine learning algorithms, statistical analysis models, and multi-dimensional analysis tools, providing them with API calls. It offers a one-stop model application service encompassing algorithm selection, model training, evaluation, and prediction, providing fundamental capabilities for scientific research data analysis, including data processing, algorithm configuration, modeling, training, evaluation, and prediction. Furthermore, to support users in using R for data analysis within the service platform, it provides the RStudio interactive analysis cloud service in a containerized manner.
[0012] This invention utilizes workflow technology to construct a research data analysis workflow application service environment. It provides research users with a unified web-based interactive interface, facilitating researchers to organize the provided basic service components—such as data preprocessing, algorithm selection, model training, evaluation, and prediction—according to the business process of data analysis and processing, and to create the required workflow instances. It supports importing and exporting the generated workflow instances, as well as publishing and sharing them among research users.
[0013] This invention mainly involves the system architecture design and construction of a scientific research data analysis model service platform, the design and development of middleware software supporting container integration, the construction of a scientific research data analysis model library, and the construction of a scientific research data analysis model workflow service engine. The overall solution of the entire method is as follows: Figure 1 As shown.
[0014] The scientific research data analysis model service platform is a collaborative environment supporting distributed deployment. It is deployed on the infrastructure of a cloud platform serving scientific research. The model management server implements the model management system, providing functions such as platform portal, project / team management, model management, publishing and sharing, and a knowledge base. Application systems such as the machine learning model system, RStudio, and R Shiny model interactive analysis system run on a cluster of backend execution servers. Users access the scientific research data analysis model service platform through a browser. Access to the model application system actually accesses the model service provided by the application container located on the execution server. The middleware software of the scientific research data analysis model service platform is responsible for container management and access routing, shielding the user from technical details; the user is unaware of which application container on which execution server actually provides the model service. This solution provides the scientific research data analysis model service platform with excellent integration capabilities and performance scalability.
[0015] This invention relates to the following parts:
[0016] 1. System Architecture of Scientific Research Data Analysis Model Service Platform. Starting from the main components such as model management server, execution server, and application container, as well as the relationships and interactions between components, a container-based model application service integration architecture is designed with the platform middleware software as the core.
[0017] 2. Platform Middleware Software. This middleware software supports the system architecture and distributed deployment of the scientific research data analysis model service platform. It primarily implements two important functions: First, it manages the lifecycle of distributed application containers. Based on user requests, it starts application containers that meet the model service requirements on appropriate execution servers, monitors the running status of these containers, and destroys them after they have completed providing the model application services. Second, after the containers start, it establishes a communication link between the user's browser and the application containers, correctly routes user requests to the application containers, and returns the responses from the application containers to the user's client.
[0018] 3. Scientific Research Data Analysis Model Library. This library integrates models and methods used in scientific research, including conventional data mining and machine learning methods (such as logistic regression, GBDT, K-Means, and support vector machines). The system provides models and algorithms such as Bayes Classifier and DRF, allowing researchers to access these tools and algorithms using languages like R, Python, and Java. These models and algorithms can be used not only in single-machine environments but also in Spark-supported big data environments, effectively supporting the analysis and processing of scientific research data.
[0019] 4. Scientific Data Analysis Workflow Engine. A scientific data analysis workflow engine is built using workflow technology. It provides research users with a unified web interface, allowing them to easily organize the provided basic service components—such as data preprocessing, algorithm selection, model training, evaluation, and prediction—according to the business process of data analysis and processing, and create the required workflow instances. It supports importing and exporting generated workflow instances, as well as publishing and sharing them among research users.
[0020] The technical solution of this invention is as follows:
[0021] A scientific research data analysis model service platform, characterized in that it includes a web server, an execution server, middleware software, and a model library;
[0022] The model library is used to store various algorithm models and application systems developed and deployed based on the algorithm models;
[0023] The middleware software includes a container execution front-end agent and a container execution back-end agent; the container execution front-end agent is located on the web server, and the container execution back-end agent is located on the execution server;
[0024] The container execution front-end proxy is used to receive user requests forwarded from the web server and perform corresponding operations according to the user requests: 1) If the user request is an application startup request, the user request information is recorded, including the user account, user terminal IP address, and the name of the application to be started; then, the corresponding execution server is selected according to the name of the application to be started by the user, and a connection is established with the container execution back-end proxy on the execution server, notifying the container execution back-end proxy to start the corresponding application container on the execution server; 2) If the user request is to monitor the status of the application container, a monitoring request is sent to the container execution back-end proxy of the execution server where the application container to be monitored is located in a periodic polling manner, the application container status is received from the container execution back-end proxy and fed back to the user; 3) If the user request is to close the application, the corresponding execution server and application container identification information are obtained according to the session information provided in the user request, and the message to close the application container is passed to the container execution back-end proxy on the corresponding execution server, notifying the container execution back-end proxy of the request to close the application container;
[0025] The container execution backend agent is used to, when receiving an application startup request sent by the container execution frontend, check whether the container image on the execution server contains an application container image that meets the requirements. If the application container image exists and the environment and conditions for running the corresponding application container are met, the agent starts the corresponding application container and returns its status to the container execution frontend agent. When receiving a listening request sent by the container execution frontend agent, the agent establishes a connection with the corresponding application container instance, obtains the running status of the application container instance, and feeds it back to the container execution frontend agent. When receiving a shutdown request sent by the container execution frontend, the agent checks the running status of the application container instance based on the session number and container identifier information provided in the shutdown request, sends a command to shut down the container instance, and then returns the shutdown status of the application container instance to the container execution frontend.
[0026] Furthermore, a web access proxy is deployed in the application container. The application receives requests from the container execution backend proxy on the execution server and forwards them to the web application deployed in the application container, and also acts as a bidirectional communication proxy between the container execution backend proxy and the web application.
[0027] Furthermore, the method for selecting the corresponding execution server based on the application name requested by the user is as follows: First, the container image of the application requested by the user exists in the container image repository of the selected execution server; if the application container image exists on multiple execution servers, the load of each execution server is calculated based on the number of application container instances running on each execution server, and the execution server with the lightest load is selected.
[0028] Furthermore, the application systems stored in the model library include statistical analysis models, ensemble learning algorithms, deep neural network algorithms, clustering algorithms, dimensionality reduction algorithms, solvers, and optimization algorithms.
[0029] A service method for the aforementioned scientific research data analysis model service platform includes the following steps:
[0030] 1) The user logs into the scientific research data analysis model service platform and sends a user request to the web server;
[0031] 2) The web server forwards the user request to the container to execute the front-end proxy, informing the user that the application system needs to be started;
[0032] 3) The container execution front-end agent locates the execution server with the corresponding application system image and passes the commands to start the container and execute container management to the container execution back-end agent located on the execution server;
[0033] 4) After the container execution backend agent confirms that the execution server it is on meets the application container runtime environment of the application system, it starts the application container of the application system, and the application container notifies the container execution backend agent that it is ready.
[0034] 5) The container execution backend agent returns a ready message to the container execution frontend agent; the container execution frontend agent accordingly modifies the URL address of the user request and redirects the user request to the container execution frontend agent located on the WEB server;
[0035] 6) The container execution front-end proxy parses the URL address of the received user request and routes the user request to the container execution back-end proxy on the execution server where the application container is running; the container execution back-end proxy parses the address of the application container according to the URL address and establishes a connection with the corresponding WEB access proxy in the application container.
[0036] 7) The WEB access proxy sends the user request to the scientific research data analysis workflow engine; the scientific research data analysis workflow engine responds to the user request and sends the response result back to the user in sequence through the WEB access proxy, the container execution backend proxy, the container execution frontend proxy, and the WEB server.
[0037] Furthermore, the scientific research data analysis workflow engine supports interactive data import, model building, model training, and model evaluation.
[0038] Furthermore, the scientific research data analysis workflow engine is a set of REST APIs built around the model application lifecycle.
[0039] Furthermore, the application systems stored in the model library include statistical analysis models, ensemble learning algorithms, deep neural network algorithms, clustering algorithms, dimensionality reduction algorithms, solvers, and optimization algorithms.
[0040] Furthermore, a web service is set up on the web server to receive user requests and return response results to the user.
[0041] Compared with the prior art, the positive effects of the present invention are as follows:
[0042] This invention proposes a method for constructing a scientific research data analysis model service platform. This method integrates and consolidates models and methods used in scientific research applications, including data mining, machine learning, and deep learning. It supports researchers in developing customized model tools and algorithms based on R, Python, Java, etc., to meet specific needs, and integrates them into the platform, effectively supporting the analysis and processing of large-scale scientific research data. The platform also integrates container technology and architecture, using container technology to achieve model componentization and service-orientation, enabling container-based model integration and services. Users can use R for data analysis within the platform, providing an interactive cloud service for RStudio in a containerized manner. A workflow engine-based model service is implemented, supporting users to import data, select algorithms, train models, predict models, and evaluate the entire scientific research data analysis process via the web. By providing a one-stop service for scientific research data analysis model applications, the platform enhances the capabilities of scientific data analysis and processing, as well as application integration, in scientific research activities, promoting knowledge sharing and innovation in scientific research collaboration models. Attached Figure Description
[0043] Figure 1 This is a schematic diagram of the overall scheme of the scientific research data analysis model service platform of this invention.
[0044] Figure 2 System architecture diagram of the platform for scientific research data analysis model service.
[0045] Figure 3 A business process diagram for starting and accessing the model service.
[0046] Figure 4 This is a system architecture diagram of a machine learning model. Detailed Implementation
[0047] To make the technical solution, creative features, objectives and effects of the present invention easier to understand, the embodiments of the present invention will be described in detail below.
[0048] To address the need for data analysis using algorithms and models in scientific research activities, this invention proposes a method for constructing a scientific research data analysis model service platform, integrating related technologies such as model algorithms, machine learning, big data processing, and cloud computing. The specific details are as follows:
[0049] 1. System Architecture of Scientific Research Data Analysis Model Service Platform
[0050] The scientific research data analysis model service platform is a collaborative environment that supports distributed deployment. The front-end web server allows users to publish, share, and access models. Application systems such as the machine learning model system, RStudio, and the R Shiny model interactive analysis system run on a cluster of back-end execution hosts. The overall architecture is as follows: Figure 2 As shown, each application system runs within a restricted, lightweight virtual environment container (i.e., the application container). This virtual environment implements access control for the file system, network, and other service processes. Each user has a separate user directory and permission control. Therefore, the sharing and access of application systems and their models are based on permission control.
[0051] The machine learning model system, RStudio, and R Shiny model interactive analysis system provide a web-based access interface and user interface. System functionality is provided through web service instances running in a container environment. Middleware software controls network operations within containers and their internal web services, authenticating requests from the user's browser and routing them to the correct application container. The middleware software mainly consists of two parts: a container execution proxy for container lifecycle management and a container web service proxy for data communication between the user's browser and the web service instances within the container. Each application container's access to and connection to external resources is limited to locally trusted services. Furthermore, the machine learning model system, RStudio, and R Shiny model interactive analysis system run in a cluster environment provided by a cloud platform serving scientific research. Therefore, based on a unified authentication and authorization mechanism, authentication and authorization management is easier, ensuring secure access to computing resources, storage resources, and visualization facilities within the research cloud platform. This facilitates system expansion and integration. The business process for users to start and access model services is as follows: Figure 3 As shown.
[0052] Taking starting and accessing a machine learning model system as an example, the entire process is as follows:
[0053] (1) After logging into the scientific research data analysis model service platform, the user requests access to the machine learning model system through a browser. Figure 3 Medium black ①);
[0054] (2) After receiving the request, the web service in the web server notifies the container located in the web server to execute the front-end proxy, informing the user that the machine learning model system (i.e., an application system in the model library) needs to be started. Figure 3 Medium black ②);
[0055] (3) The container execution front-end agent locates the execution server with the machine learning model container image and passes the commands to start the container and manage the execution container to the container execution back-end agent located on the execution server. Figure 3 (Middle black ③);
[0056] (4) After the container execution backend agent confirms that the execution server meets the environment requirements for running the machine learning model system application container, it starts the machine learning model system application container. Figure 3 After completing initialization settings, starting the Web application service, and starting the Web access proxy, the application container informs the container execution backend proxy machine learning model system application container located on the same execution server that it is ready. Figure 3 (Middle black ⑤);
[0057] (5) The container execution backend agent returns a ready message to the container execution frontend agent. Figure 3 (Black in the middle ⑥), the container executes the front-end proxy to modify the URL request from the user's browser accordingly, and redirects the modified URL request to the container web service front-end proxy located on the web server. Figure 3 Medium black (⑦);
[0058] (6) The container web service front-end proxy parses the received URL request and correctly routes the URL request to the container web service back-end proxy on the execution server where the application container is running. Figure 3 (Middle Black ⑧). Because multiple application containers may be running simultaneously on the execution server, the container web service backend proxy resolves the application container's address based on the request URL and establishes a connection with the web access proxy within the application container. Figure 3 (Black 9)
[0059] (7) The WEB access proxy parses and translates the request URL, establishing a connection with the machine learning model WEB application service. This machine learning model WEB application service is the scientific data analysis workflow engine, which will be introduced later. Its core is a set of REST APIs built around the model application lifecycle, and a web-based interactive computing environment provided on top of the workflow engine. It supports interactive data import, model building, model training, and model evaluation. The request is then passed to the machine learning model WEB application service. After the WEB application service responds, the response content is returned to the user's browser based on the previously established connection, sequentially through the WEB access proxy, the container WEB service backend proxy, the container WEB service frontend proxy, and the WEB service on the WEB server. Figure 3 Medium Black 10 ).
[0060] (8) After the application container web service access session is established, subsequent interactive access only requires pressing the button. Figure 3 The path is traversed using the solid line in the middle.
[0061] In terms of model management, publishing, and sharing, it supports the organization and management of models at both the user and project levels. It also supports the further development of a collaborative research data analysis platform centered on model management and sharing.
[0062] 2. Platform Middleware Software
[0063] To support the system architecture and distributed deployment of the scientific research data analysis model service platform, the platform's middleware software primarily implements two important functions. The first is the lifecycle management of distributed application containers. This involves launching application containers that meet the model service requirements on appropriate execution servers based on user requests, monitoring the operational status of these containers, and destroying the containers after they have completed providing the model application services. This functionality is mainly achieved through two components: a container execution front-end proxy and a container execution back-end proxy.
[0064] The container executes a front-end proxy. It receives user requests forwarded from the model management server's web service. Acting as an intermediary proxy between the model management server and the execution server, it performs corresponding operations based on the user requests. The container executes a front-end proxy, primarily supporting three types of operations, as detailed below:
[0065] (1) Receive application startup requests from users. Record user request information, including user account, user IP address, and the name of the requested application. Then, based on the requested application name, select an appropriate execution server, establish a connection with the container execution backend agent on that execution server, and notify the container execution backend agent to start the corresponding application container on the execution server. The key to the whole process is selecting an appropriate execution server based on the user request. The selection of the execution server mainly considers two factors. First, the container image repository of the selected execution server contains the application container image requested by the user; second, when multiple execution servers contain the application container image, the load of each execution server is calculated based on the number of application container instances running on the current execution server, and the execution server with the lightest load is selected as the target. This achieves a certain degree of load balancing and ensures the scalability and stability of the entire system platform.
[0066] (2) Monitoring the application container's status. After the application container starts, throughout the entire session before the user stops the application and destroys the application container, the container execution backend agent periodically polls the application container's Heartbeat status. This means it periodically sends listening requests to the container execution backend agent on the execution server where the application container resides, receives the application container status feedback from the container execution backend agent, and then reports it back to the user. The application container remains running until the user explicitly stops it. Therefore, the user can close the client browser and reconnect to the previously started application container on the next visit to continue their work.
[0067] (3) Receive a request from the user to shut down the application. Based on the session information provided in the user's request, obtain the corresponding execution server and application container identification information, and pass the message to shut down the application container to the container execution backend agent on the corresponding execution server, notifying the other party of the request to shut down the application container. Receive the application container shutdown status information transmitted back by the container execution backend agent of the execution server.
[0068] The container execution backend proxy. This proxy runs as a service on the execution server. It acts as an intermediary between the container execution frontend proxy and the application container, performing corresponding operations. Details are as follows:
[0069] (1) Receive the application startup request sent by the container execution frontend and check whether the container images on this execution server contain an application container image that meets the requirements. If this execution server has the environment and conditions to run the application container corresponding to the application container image, then start the application container and return the startup status of the application container to the container execution frontend agent. The startup of the application container mainly includes two processes: first, the container runtime environment is built, including file system loading, network environment settings, etc.; second, the application-related services are started, such as starting a web service application or a web service proxy within the container instance.
[0070] (2) Monitor the status of the application container. When a monitoring request is received from the container execution front-end agent, a connection is established with the application container instance to obtain the running status of the application container instance and return it to the container execution front-end agent.
[0071] (3) Receive the application shutdown request sent by the container execution frontend. Based on the session number, container identifier, and other information provided in the request, check the running status of the application container instance and send a command to shut down the container instance. Check the shutdown status of the container instance and record the resource usage of the application container instance. Finally, return the shutdown status of the application container instance to the container execution frontend.
[0072] The second function of the platform middleware software is to establish a communication link between the user's browser and the application container after the container starts, correctly route user requests to the application container, and return the application container's response to the user. In the middleware software, this function is mainly implemented through three components: the container web service front-end proxy, the container web service back-end proxy, and the web access proxy.
[0073] Container Web Service Front-End Proxy. The container web service front-end proxy can be deployed on the model management server or a separate server. Its function is to receive requests from the user's browser and establish a connection channel between the user's browser and the container web service back-end proxy, enabling bidirectional communication. Specifically, the container web service front-end proxy receives HTTP(S) requests from the user's browser, parses the session identifier and container identifier of the previously established container session from the URL, obtains the IP address and communication port of the execution server through the session identifier, creates a network communication socket, establishes a connection with the container web service back-end proxy located on the execution server, and supports subsequent bidirectional communication.
[0074] Container Web Service Backend Proxy. The container web service backend proxy is deployed on the execution server. Its function is to receive requests from the container web service frontend proxy, locate the application container instance based on the container identifier in the URL, connect to the web access proxy located inside the container based on the application container instance's IP address and port, and then act as a bidirectional communication proxy between the container web service frontend proxy and the web access proxy.
[0075] A web access proxy is deployed within an application container. It receives requests from the container web service backend proxy on the execution server and forwards them to the web application deployed within the container—that is, the application system or model algorithm in the model library, such as machine learning model systems, RStudio, and the R Shiny model interaction analysis system. It then acts as a proxy for bidirectional communication between the container web service backend proxy on the execution server and the web application within the container.
[0076] 3. Scientific research data analysis model library
[0077] This system integrates and consolidates research application models and methods to meet the needs of the scientific research field, including conventional data mining, machine learning, and deep learning methods. It supports researchers in developing model-based research applications using R, Python, Java, and other languages. The constructed machine learning model system is deeply integrated with big data computing platform environments such as Hadoop and Spark, enabling distributed access and sharing of big data. It supports MapReduce and traditional Fork / Join methods for big data analysis and processing. Utilizing the memory-based distributed parallel processing capabilities provided by the machine learning model system, model training and prediction calculations are submitted to the big data analysis and processing platform for efficient computation. The machine learning model system structure is as follows: Figure 4 As shown.
[0078] The scientific research data analysis model library integrates mainstream machine learning algorithms, as follows:
[0079] (1) Statistical analysis, providing linear models, including linear regression, logistic regression, and other linear regression and classification models, as well as Naive Bayes classification. BayesClassifier, Support Vector Machine (SVM), etc.
[0080] (2) Ensemble learning, such as decision tree, distributed random forest (DRF), gradient boosting machine (GBM), etc.
[0081] (3) Deep neural network class, supporting multi-layer feed-forward neural network, auto-encoder, anomaly detection, etc.;
[0082] (4) Clustering, providing K-means clustering;
[0083] (5) Dimension reduction class, providing Principal Component Analysis (PCA) and Generalized Low Rank Models (GLRM);
[0084] (6) Solver & Optimization Classes: Provides Generalized ADMM Solver, Quasi Newton Method (L-BFGS), Ordinary Least-Square Solver, and Stochastic Gradient Descent.
[0085] The list of algorithms supported by the scientific research data analysis model library is shown in Table 1.
[0086] Table 1. List of algorithms supported by the scientific research data analysis model library
[0087]
[0088]
[0089] User-trained models can be compiled into Java objects and easily embedded into any Java environment. A web-based user interface is provided, supporting users in importing data, selecting algorithms, training models, predicting models, and evaluating them via the web. Application programming interfaces (APIs) for languages such as R, Python, and Java are also provided.
[0090] 4. RStudio and R Shiny model interactive analysis system
[0091] The scientific research data analysis model service platform provides RStudio and the R Shiny model interactive analysis system, integrating statistical and data analysis functions based on R. The platform deploys the web version of the RStudio IDE into an application container, utilizing container technology to achieve application space isolation. The same mechanism applies to applications built on Shiny (a software package for building web applications based on R) and RMarkdown-based documents. Similar to RStudio IDE, Shiny applications and RMarkdown documents require a web service running within the application container to receive requests from the user's browser routed by middleware software. The application container provides application isolation and corresponding security policies for these web applications. A data statistical analysis system built on R and Shiny enables dataset loading, previewing, statistical analysis, and pivot table analysis, supporting data operations such as dataset changes, merging, and simple visualization.
[0092] 5. Scientific research data analysis workflow engine.
[0093] The core of the scientific research data analysis workflow engine is a set of REST APIs built around the model application lifecycle. It provides REST APIs supporting five main categories of operations: data import / export, algorithm selection, model training, model prediction and evaluation, and model import / export.
[0094] The model application workflow is represented as a set of executable operation units organized in chronological order, which can be edited and sorted. The entire workflow is organized in JSON format and can be saved as a workflow file, recording the order of the operation units and the corresponding parameter settings for each operation. Models and model application processes can be distributed and shared by sharing workflow files.
[0095] Building upon the workflow engine, a web-based interactive computing environment is provided, allowing users to interactively import data, build models, and then iterate to improve them, storing the entire process as a workflow file. Simultaneously, users can import and parse workflow files, and based on the parsed operation units, automatically or interactively call a REST API to execute model application operations, completing the workflow.
[0096] This invention proposes a method for constructing a scientific research data analysis model service platform. The model service platform constructed by this method integrates and consolidates models and methods used in scientific research applications, including conventional data mining, machine learning methods, and deep learning methods; it supports users in defining and validating their own developed or customized models. It provides functions including model definition, model training, model validation, and model evaluation; and enables lifecycle management of scientific research data analysis models. It also supports the online publication and sharing of scientific research data analysis models and algorithms.
[0097] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.
Claims
1. A scientific research data analysis model service platform, characterized in that, This includes web servers, execution servers, middleware software, and model libraries; The model library is used to store various algorithm models and application systems developed and deployed based on the algorithm models; The middleware software includes a container execution front-end agent and a container execution back-end agent; the container execution front-end agent is located on the web server, and the container execution back-end agent is located on the execution server; The container execution front-end proxy is used to receive user requests forwarded from the web server and perform corresponding operations according to the user requests: 1) If the user request is an application startup request, the user request information is recorded, including the user account, user terminal IP address, and the name of the application to be started; then, the corresponding execution server is selected according to the name of the application to be started by the user, and a connection is established with the container execution back-end proxy on the execution server, notifying the container execution back-end proxy to start the corresponding application container on the execution server; 2) If the user request is to monitor the status of the application container, a monitoring request is sent to the container execution back-end proxy of the execution server where the application container to be monitored is located in a periodic polling manner, the application container status is received from the container execution back-end proxy and fed back to the user; 3) If the user request is to close the application, the corresponding execution server and application container identification information are obtained according to the session information provided in the user request, and the message to close the application container is passed to the container execution back-end proxy on the corresponding execution server, notifying the container execution back-end proxy of the request to close the application container; The container execution backend agent is used to, when receiving an application startup request sent by the container execution frontend, check whether the container image on the execution server contains an application container image that meets the requirements. If the application container image exists and the environment and conditions for running the corresponding application container are met, the agent starts the corresponding application container and returns its status to the container execution frontend agent. It also establishes a connection with the corresponding application container instance when receiving a listening request from the container execution frontend agent, obtains the running status of the application container instance, and feeds it back to the container execution frontend agent. Furthermore, when receiving a shutdown request from the container execution frontend, it checks the running status of the application container instance based on the session number and container identifier information provided in the shutdown request, and sends a command to shut down the container instance. Then, the application container instance is returned to the container execution frontend as closed.
2. The scientific research data analysis model service platform according to claim 1, characterized in that, A web access proxy is deployed in the application container. The application receives requests from the container execution backend proxy on the execution server and forwards them to the web application deployed in the application container, and also acts as a bidirectional communication proxy between the container execution backend proxy and the web application.
3. The scientific research data analysis model service platform according to claim 1, characterized in that, The method for selecting the corresponding execution server based on the application name requested by the user is as follows: First, the container image of the application requested by the user exists in the container image repository of the selected execution server; if the application container image exists on multiple execution servers, the load of each execution server is calculated based on the number of application container instances running on each execution server, and the execution server with the lightest load is selected.
4. The scientific research data analysis model service platform according to claim 1, 2, or 3, characterized in that, The application systems stored in the model library include statistical analysis models, ensemble learning algorithms, deep neural network algorithms, clustering algorithms, dimensionality reduction algorithms, solvers, and optimization algorithms.
5. A service method based on the scientific research data analysis model service platform of claim 1, comprising the following steps: 1) The user logs into the scientific research data analysis model service platform and sends a user request to the web server; 2) The web server forwards the user request to the container to execute the front-end proxy, informing the user that the application system needs to be started; 3) The container execution front-end agent locates the execution server with the corresponding application system image and passes the commands to start the container and execute container management to the container execution back-end agent located on the execution server; 4) After the container execution backend agent confirms that the execution server it is on meets the application container runtime environment of the application system, it starts the application container of the application system, and the application container notifies the container execution backend agent that it is ready. 5) The container execution backend agent returns a ready message to the container execution frontend agent; The container execution front-end proxy accordingly modifies the URL address of the user request and redirects the user request to the container execution front-end proxy located on the WEB server; 6) The container execution front-end proxy parses the URL address of the received user request and routes the user request to the container execution back-end proxy on the execution server where the application container is running; the container execution back-end proxy parses the address of the application container according to the URL address and establishes a connection with the corresponding WEB access proxy in the application container. 7) The WEB access proxy sends the user request to the scientific research data analysis workflow engine; the scientific research data analysis workflow engine responds to the user request and sends the response result back to the user in sequence through the WEB access proxy, the container execution backend proxy, the container execution frontend proxy, and the WEB server.
6. The method according to claim 5, characterized in that, The scientific research data analysis workflow engine supports interactive data import, model building, model training, and model evaluation.
7. The method according to claim 6, characterized in that, The scientific research data analysis workflow engine is a set of REST APIs built around the lifecycle of model application.
8. The method according to claim 5, characterized in that, The application systems stored in the model library include statistical analysis models, ensemble learning algorithms, deep neural network algorithms, clustering algorithms, dimensionality reduction algorithms, solvers, and optimization algorithms.
9. The method according to claim 5, characterized in that, A web service is set up on the web server to receive user requests and return response results to the user.