Algorithmic pipeline model construction system, optimization system, and method for software systems

Abstract

The present disclosure provides an algorithm pipeline model construction system, optimization system and method for a software system, which is based on the form of interface visualized drag-and-drop arrangement components to construct an algorithm pipeline model, and runs algorithm models and data conversion methods in a container manner in an externally closed environment, thereby reducing the possibility of conflicts of related packages; meanwhile, component parameters can be automatically adjusted according to the version of the algorithm pipeline model, thereby reducing the time cost and complexity of parameter configuration, and improving the reusability of the algorithm pipeline model through a common template.

Description

Technical Field

[0001] This disclosure belongs to the field of data flow processing technology, and in particular relates to an algorithm pipeline model construction system, optimization system and method for software systems. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Intelligent analysis systems typically employ intelligent algorithms to analyze and process data, presenting the results to users in an appropriate manner. The main tasks of an intelligent analysis system include data acquisition, data preprocessing, data analysis and mining, and result visualization. Implementing an intelligent analysis system involves designing a data processing workflow to determine the collaborative processes between various tasks. Data processing involves extracting data from a wide range of heterogeneous data sources, storing them uniformly according to certain standards, using data analysis techniques to extract useful knowledge, and presenting the results to users. Knowledge is often extracted through an algorithmic pipeline, which is an ordered process combining steps such as data preprocessing, algorithm model execution, and data post-processing. Building an algorithmic pipeline model involves combining these steps to form a complete data processing and analysis workflow.

[0004] The key steps in building an existing algorithm pipeline model are: designing data preprocessing and postprocessing by writing code; and designing algorithm model execution by calling local or algorithm service provider algorithm interfaces.

[0005] The inventors discovered that the construction of existing algorithm pipeline models has the following problems: once the code for data preprocessing, postprocessing, and calling algorithm interfaces is encapsulated into the algorithm pipeline, the components inside the pipeline are difficult to reuse; the related packages of locally deployed algorithm models have strong version dependencies, and packages running in the same environment are prone to version conflicts; calling the algorithm interfaces of algorithm service providers still requires configuring the runtime environment of different providers, and calling the algorithm interfaces of multiple algorithm service providers at the same time is prone to runtime environment conflicts.

[0006] Furthermore, after constructing an algorithm pipeline model, it is often necessary to continuously optimize the parameters of each component within the model to ensure optimal performance in real-world scenarios. Current methods for optimizing the parameters of each component in an algorithm pipeline model primarily rely on the experience and knowledge of engineers / implementers, employing a trial-and-error approach to find the optimal parameters, which is inefficient. Once the parameters of data preprocessing, post-processing, and the algorithm model are fixed within the algorithm pipeline, the internal parameters cannot adaptively change according to the application scenario, leading to inaccurate data analysis results from the constructed algorithm pipeline model. Summary of the Invention

[0007] To address the aforementioned issues, this disclosure provides an algorithm pipeline model construction system, optimization system, and method for software systems. The solution constructs the algorithm pipeline model based on a visual drag-and-drop component arrangement interface, running the algorithm model and data transformation methods in a containerized, closed environment, reducing the possibility of conflicts between related packages. Simultaneously, component parameters can be automatically adjusted according to the version of the algorithm pipeline model, reducing the time cost and complexity of parameter configuration, and improving the reusability of the algorithm pipeline model through common templates.

[0008] According to a first aspect of the present disclosure, an algorithm pipeline model construction system for a software system is provided, comprising:

[0009] An algorithm model and data conversion method deployment module is used to receive the algorithm model, data conversion method, environment dependency information, and parameter information to be deployed, and to build and deploy the containers corresponding to the algorithm model and data conversion method; wherein, the algorithm model and data conversion method are represented in the form of container components;

[0010] The container information proxy module is used to convert container information from the algorithm model and data transformation method deployment module into component properties and forward them to the component storage module; and, based on the container's state, to request in real time whether the component in the component storage module has enabled property modification.

[0011] The component storage module is used to store container components and maintain component attribute information based on requests from the container information proxy module; as well as to store algorithm pipeline models, task execution sequences, and common templates.

[0012] The component orchestration module is used to orchestrate the components in the component storage module using a graphical drag-and-drop interface, thereby building the algorithm pipeline model.

[0013] Furthermore, the system also includes a component visualization module and a template generation module. The visualization unit is used to visualize the components and common templates within the component storage module. The template generation unit is used to generate corresponding common templates and store them in the component storage module based on the algorithm pipeline model obtained from the component orchestration results through serialization processing.

[0014] Furthermore, the deployment of the algorithm model and data conversion method specifically involves: encapsulating the received executable files of the algorithm model and data conversion method to be deployed into service files and copying them to a specified working directory; and automatically creating and deploying a container image of the algorithm model and data conversion method.

[0015] Furthermore, the algorithm model is deployed as an algorithm capsule component in the form of an algorithm capsule. The algorithm capsule includes external attributes and an internal structure. The external attributes include the trained algorithm model source code and the external dependencies of the algorithm capsule. The internal structure includes an input / output interface layer, a core layer, and an environment dependency layer built based on the external attributes. The core layer is used to preprocess the input data of the current algorithm model's corresponding task according to its corresponding application domain. Based on the preprocessed input data, the algorithm model's data processing service is requested to obtain the processing result.

[0016] Furthermore, the task execution sequence specifically refers to the execution order of each processing step in the algorithm pipeline model.

[0017] According to a second aspect of the present disclosure, an algorithm pipeline model optimization system for a software system is provided, which is applied to optimize the algorithm pipeline model constructed by the aforementioned algorithm pipeline model construction system for a software system, including:

[0018] The algorithm pipeline parsing module is used to parse the algorithm pipeline model to be optimized and obtain the task execution sequence of the algorithm pipeline model based on the parsing results;

[0019] The algorithm pipeline execution module is used to calculate the weighted sum of evaluation metrics of the current algorithm pipeline model based on the obtained task execution sequence of the algorithm pipeline model; and to optimize the parameters of the algorithm pipeline model according to the difference between the weighted sum of evaluation metrics and the expected threshold, so as to obtain the optimized algorithm pipeline model; wherein, in the optimization process, different versions of the algorithm pipeline model are obtained and stored.

[0020] The model information display and version switching module is used to display different versions of the algorithm pipeline model and automatically configure the parameters of the corresponding version of the algorithm pipeline model based on the selected version, so as to realize the version switching of the algorithm pipeline model.

[0021] Furthermore, the optimization of the parameters of the algorithm pipeline model specifically involves using a bisection method to continuously correct the parameters of the algorithm pipeline model based on the changing trend of the results during the optimization process.

[0022] Furthermore, the evaluation metrics include, but are not limited to, the accuracy of classification tasks, the mean squared error of regression tasks, and the precision of recommendation tasks.

[0023] According to a third aspect of the present disclosure, a method for constructing an algorithm pipeline model for a software system is provided, which is based on the aforementioned algorithm pipeline model construction system for a software system, the method comprising:

[0024] The module is based on the algorithm model and data transformation method deployment to build and deploy containers corresponding to the algorithm model and data transformation method.

[0025] The container information proxy module converts container information into component properties and forwards them to the component storage module; and, based on the container's state, it requests in real time whether the component in the component storage module has enabled property modification.

[0026] The component storage module stores container components and maintains component attribute information based on requests from the container information proxy module; it also stores algorithm pipeline models, task execution sequences, and common templates.

[0027] Based on the component orchestration module, the components in the component storage module are orchestrated using a graphical drag-and-drop interface to build the algorithm pipeline model.

[0028] According to a fourth aspect of the present disclosure, a method for optimizing an algorithm pipeline model for a software system is provided, comprising:

[0029] The algorithm pipeline model to be optimized is analyzed, and the task execution sequence of the algorithm pipeline model is obtained based on the analysis results;

[0030] Based on the obtained task execution sequence of the algorithm pipeline model, the weighted sum of the evaluation metrics of the current algorithm pipeline model is calculated; and the parameters of the algorithm pipeline model are optimized according to the difference between the weighted sum of the evaluation metrics and the expected threshold to obtain the optimized algorithm pipeline model; wherein, in the optimization process, different versions of the algorithm pipeline model are obtained and stored.

[0031] Different versions of the algorithm pipeline model are displayed, and the parameters of the corresponding version of the algorithm pipeline model are automatically configured based on the selected version, so as to realize the version switching of the algorithm pipeline model.

[0032] Compared with the prior art, the beneficial effects of this disclosure are:

[0033] (1) This disclosure provides an algorithm pipeline model construction system, optimization system and method for software systems. The solution constructs the algorithm pipeline model based on the form of interface visual drag-and-drop arrangement components, and runs the algorithm model and data transformation method in a containerized manner in an externally closed environment, which reduces the possibility of conflicts between related packages. At the same time, the component parameters can be automatically adjusted according to the version of the algorithm pipeline model, which reduces the time cost and complexity of parameter configuration, and improves the reusability of the algorithm pipeline model through the use of common templates.

[0034] (2) Compared with the existing methods of optimizing algorithm pipeline model parameters, the scheme described in this disclosure improves the efficiency of finding the optimal model parameters by calculating the trend of the difference between the weighted sum of the evaluation index and the expected effect, and continuously adjusting the parameters using the bisection method. At the same time, the scheme provides recommended parameter configurations according to different scenarios through the information display and version switching modules, fills the algorithm pipeline model with different parameter configurations and switches between different versions, thereby improving the ability of the algorithm pipeline model to adapt to different application scenarios.

[0035] Advantages of this disclosure in additional aspects will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0036] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0037] Figure 1 This is a schematic diagram of the overall processing flow of an algorithm pipeline model construction system for a software system as described in an embodiment of this disclosure;

[0038] Figure 2 This is a schematic diagram illustrating the process of creating an image and deploying it to a container as described in the embodiments of this disclosure;

[0039] Figure 3 This is a flowchart illustrating the container information proxy module described in this embodiment of the disclosure;

[0040] Figure 4 This is a flowchart illustrating the automatic optimization method for algorithm pipeline model parameters described in the embodiments of this disclosure;

[0041] Figure 5 This is a flowchart illustrating the parsing algorithm pipeline model and the process of obtaining the task execution sequence as described in the embodiments of this disclosure.

[0042] Figure 6 This is a schematic diagram of the algorithm pipeline execution process described in the embodiments of this disclosure. Detailed Implementation

[0043] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0045] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0046] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0047] Terminology Explanation:

[0048] Algorithm capsule: A container instance for deploying and implementing algorithmic models, called an algorithm capsule, which provides data processing services for the algorithmic model as an independent unit.

[0049] Example 1:

[0050] The purpose of this embodiment is to provide an algorithm pipeline model construction system for software systems.

[0051] A system for constructing algorithm pipeline models for software systems, comprising:

[0052] An algorithm model and data conversion method deployment module is used to receive the algorithm model, data conversion method, environment dependency information, and parameter information to be deployed, and to build and deploy the containers corresponding to the algorithm model and data conversion method; wherein, the algorithm model and data conversion method are represented in the form of container components;

[0053] The container information proxy module is used to convert container information from the algorithm model and data transformation method deployment module into component properties and forward them to the component storage module; and, based on the container's status, to request in real time whether the component in the component storage module has enabled property modification.

[0054] The component storage module is used to store container components and maintain component attribute information based on requests from the container information proxy module; as well as to store algorithm pipeline models, task execution sequences, and common templates.

[0055] The component orchestration module is used to orchestrate the components in the component storage module using a graphical drag-and-drop interface, thereby building the algorithm pipeline model.

[0056] In a specific implementation, the system further includes a component visualization module and a template generation module. The visualization unit is used for the visual display of components and common templates within the component storage module. The template generation unit is used to generate corresponding common templates based on the algorithm pipeline model obtained from the component orchestration results, through serialization processing, and store them in the component storage module.

[0057] In specific implementation, the deployment of the algorithm model and data conversion method involves: encapsulating the executable files of the received algorithm model and data conversion method to be deployed into service files and copying them to the specified working directory; and automatically creating and deploying container images of the algorithm model and data conversion method.

[0058] In specific implementation, the algorithm model is deployed as an algorithm capsule component in the form of an algorithm capsule. The algorithm capsule includes external attributes and internal structure. The external attributes include the trained algorithm model source code and the external dependencies of the algorithm capsule. The internal structure includes an input / output interface layer, a core layer, and an environment dependency layer built based on the external attributes. The core layer is used to preprocess the input data of the current algorithm model's corresponding task according to its corresponding application domain. Based on the preprocessed input data, the algorithm model's data processing service is requested to obtain the processing result.

[0059] In specific implementation, the task execution sequence refers to the execution order of each processing step in the algorithm pipeline model.

[0060] Specifically, for ease of understanding, the following detailed description of the solution in this embodiment is provided in conjunction with the accompanying drawings:

[0061] A system for constructing algorithm pipeline models for software systems, comprising:

[0062] The algorithm model and data transformation method deployment module is responsible for receiving the executable file of the algorithm model and data transformation method, the environment dependency information of the executable file, and the parameter information of the algorithm model and data transformation method. It copies the executable file to the specified working directory and automatically creates and deploys the image script file of the algorithm model and data transformation method. This device communicates with the container information agent module to transmit container parameters and port information.

[0063] The container information proxy module is responsible for receiving container information and processing it into a list containing attributes such as container number, name, type, parameters, whether it is enabled, and container address, before forwarding it to the component storage module. It is also responsible for periodically detecting changes in container status and requesting the component storage module to change the enable / disable attribute of the algorithm capsule component or data transformation component in real time. The container status in the container information proxy module includes created, running, paused, exited, dead, healthy, and unhealthy.

[0064] The component storage module receives requests from the container information proxy module. It stores and maintains in real-time the attributes of each algorithm capsule component and data transformation component, including their ID, name, type, parameters, whether they are enabled, and container address. The component storage module also supports storing and maintaining attributes of the algorithm pipeline model, task execution sequences, and common templates. The task execution sequence is a series of tasks or operations executed in a specific order, reflecting the execution order of each step in the algorithm pipeline. The task execution sequence includes four attributes: task ID, task name, container address sequence, and parameter sequence. The common template includes six attributes: template ID, template name, template version, template icon, component ID sequence, and component constraint sequence.

[0065] The component visualization module is responsible for accessing the properties of each component and common template in the component storage module. It visualizes the components and templates through a graph editing engine and supports grouping, collapsing, searching, dragging and dropping.

[0066] The component orchestration module is responsible for arranging and combining components and configuring their parameters. It generates visual connection endpoints for each component based on the number of parameters in its constraint attributes. The upper endpoint of a component represents the data flow entry point, and the lower endpoint represents the data flow exit point. During orchestration, a parameter configuration panel is assigned to each component, with the configuration items corresponding to the parameter information in the component's constraints. These configuration items support basic data format validation and can be filled with content through various configuration methods, such as input boxes, dropdown lists, single / multiple selections, dates, and image file formats. The component orchestration module communicates with the template generation module, transmitting the orchestration result's name, icon, type, version, constraints, component connection relationships, and configuration information.

[0067] The template generation module is responsible for receiving the orchestration results from the component orchestration module, serializing the orchestrated algorithm pipeline model, and converting it into a JSON-formatted text object; extracting the component connection relationships and configuration information from the text object and converting them into component number sequences and component constraint sequences, which are then passed to the component storage module as attributes of the newly added common template; and requesting the component visualization module to visualize the common template.

[0068] The algorithm model is deployed as an algorithm capsule component in the form of an algorithm capsule. The algorithm capsule includes external attributes and internal structure. The external attributes include the trained algorithm model source code and the external dependencies of the algorithm capsule. The internal structure includes an input / output interface layer, a core layer, and an environment dependency layer built based on the external attributes. The core layer is used to preprocess the input data of the current algorithm model's corresponding task according to its corresponding application domain. Based on the preprocessed input data, it requests the data processing service of the algorithm model to obtain the processing result.

[0069] The external attributes of the algorithm capsule mainly include environment dependencies, core code, command set, algorithm type, input / output interface, etc.

[0070] The input and output interfaces define how the algorithm capsule interacts with the external environment. Their main attributes include file format, data type, input data source, and output data target. The file format refers to the file format of the algorithm's input and output data, primarily including JSON, XML, and CSV formats. The data type refers to the type of data processed by the algorithm, including image, text, and audio data. The input data source types mainly include files, databases, network interfaces, or other algorithm capsules. The output data target types mainly include files, databases, network interfaces, or other algorithm capsules.

[0071] The algorithm types include task types and application domain types. The task type refers to the task the algorithm model aims to solve, including classification, regression, clustering, association rule analysis, etc. The application domain type refers to the specific field where the algorithm model is used to solve real-world problems, including healthcare, financial services, intelligent transportation, sales, agriculture, education, and other fields.

[0072] The command set is a collection of run commands and control commands. Run commands control the execution state of the algorithm capsule, including start, stop, and destroy commands. The start command starts the runtime environment of the algorithm model and loads its dependencies; the stop command stops the algorithm capsule and closes exposed service ports; the destroy command deletes the algorithm capsule and releases its resources. Control commands control the data transmission of the algorithm capsule, including request and response commands. Request commands send data to the algorithm model and request data processing services. Response commands receive the results returned by the algorithm model and send them to the output interface.

[0073] The core code is a pre-trained algorithm model that provides data processing services. An algorithm model refers to a predictive model built using mathematical methods based on existing data, which is then used for predicting and classifying unknown data. Alternatively, it can utilize deep neural network structures, supported by large-scale data and computing power, to learn and construct multi-level feature representations from input data, thus building an algorithm model.

[0074] The environment dependencies include the runtime environment and dependencies. The runtime environment includes the operating system, compilation and execution middleware, CPU and GPU computing platforms, etc. The dependencies are the library files necessary for the algorithm to run.

[0075] The algorithm capsule's internal structure, from top to bottom, consists of an input / output interface layer, a core layer, and an environment dependency layer. External attributes refer to the attribute information that describes or defines the algorithm capsule.

[0076] The internal structure refers to the functional structure and operating mode within the algorithm capsule.

[0077] The input / output interface layer serves as the interface between the algorithm capsule and the external environment. The input interface receives data from the external environment and converts it into a type that the algorithm model can process through the file conversion module. In real-world applications, the algorithm capsule receives a wide variety of data types, such as text, audio, and images. However, each algorithm model can only handle a single data type, so the input interface converts the received data into a data type that the algorithm model can process. The data conversion module calls the corresponding data processing methods to complete the conversion of received image and audio data, then saves the data in the text file format specified by the input interface, and finally passes the text file to the core layer.

[0078] The core layer's data preprocessing layer preprocesses data according to application domain type. Based on the application domain corresponding to the preprocessing module, it calls data preprocessing methods to perform data preprocessing operations such as standardizing data format, standardizing domain-specific terms, deleting redundant data, and filling in missing data values.

[0079] The core layer's control commands pass the preprocessed data to the algorithm model and request the algorithm model's data processing services. The request command in the control command includes a parameter indicating the type of task requested, which specifies the type of task the algorithm model needs to solve.

[0080] The core layer's algorithm model receives data from the request command, calls the algorithm to process the data based on the request task type parameter, obtains the processing result, and returns the result to the control command's response command. The response command packages the algorithm model's processing result and passes it to the output interface of the input / output interface layer. The output interface saves the data processing result in a specified file format and then sends the file to the specified output data target.

[0081] The run command can control the running status of each internal process, including start, stop, and destroy commands.

[0082] The environment dependency layer provides the internal environment of the algorithm capsule with support for operating system, programming environment, and high-performance computing, process scheduling, real-time performance monitoring, and file system access for the algorithm model.

[0083] The implementation of the above system mainly adopts the following technical concept: First, the algorithm model and data transformation method are converted into images, and then a runnable container is created. The container's runtime environment is isolated from the external environment, solving the version dependency problem of related packages. Second, the data format and parameter information required by the algorithm capsule and data transformation are mapped into component attributes, further visualizing the components. The algorithm capsule is a container for deploying the algorithm model. The data transformation is a series of operations performed before and after executing the algorithm, such as data cleaning, data extraction, text classification, and anomaly detection. Third, the algorithm capsule components and data transformation components are orchestrated, and the component attributes are configured to obtain the algorithm pipeline model. The algorithm capsule components and data transformation components are respectively encapsulations of the algorithm capsule and data transformation method. Finally, the relevant components contained in the algorithm pipeline model are used as common templates to facilitate quick switching and reuse of the algorithm pipeline model. The common templates are general components defined to enable the rapid construction of algorithm pipeline models.

[0084] Simultaneously, after constructing the algorithm pipeline model, it is necessary to optimize the parameters of each component within the model. Specifically, the following strategy is adopted: based on the changing trends of the results during the optimization process, a binary search method is used to continuously correct the algorithm pipeline model parameters, thereby enabling the algorithm model to achieve optimal performance in real-world scenarios. Furthermore, recommended parameter versions are provided for different scenarios, allowing for switching between different algorithm pipeline model versions by filling in different parameter configurations.

[0085] The algorithm pipeline model includes six attributes: pipeline number, pipeline name, pipeline version, component number sequence, component constraint sequence, and evaluation metric sequence. The evaluation metrics are used to measure the performance and effectiveness of the algorithm pipeline model, and their values ​​vary depending on the algorithm parameters and application scenario. The evaluation metric sequence is a sequence composed of evaluation metrics for different application scenarios, such as accuracy for classification tasks, mean squared error for regression tasks, and precision for recommendation systems.

[0086] Each component includes eight attributes: component number, component name, component type, component version, component icon, component constraints, whether enabled, and container address. The component constraints contain information such as the range, values, and variation rules of the parameters of the specific algorithm or method represented by the component.

[0087] like Figure 1 As shown, in practical applications, the system specifically executes the following process:

[0088] Step S101: Submit the executable file, parameter configuration, environment dependencies, and other information of the algorithm model and data transformation method to the algorithm model and data transformation method deployment module. The algorithm model and data transformation method deployment module creates an image script file. It builds a container image and deploys it to the container, while configuring port mapping rules to map the ports listened to by the algorithm model and data transformation method inside the container to externally accessible ports. Finally, it submits the externally accessible ports and parameter configuration information to the container information broker module.

[0089] In step S101, the algorithm model and data transformation method deployment module creates an image and deploys it to the container as follows: Figure 2 As shown, the specific steps are as follows:

[0090] First, the image script file includes the base image, working directory, environment dependencies, and configuration parameters. It encapsulates the algorithm model and data transformation methods using a web framework, providing a REST interface for execution as a service. The parameters of the REST interface are packaged into configuration parameter instructions. The frameworks that the algorithm model service depends on are packaged into the image script's configuration base image instructions; the programming environments of the methods in the data transformation service are also packaged into the image script's configuration base image instructions. Second, the service files are copied to the specified working directory, and the path is packaged into the image script's configuration working directory instructions. Third, the package names and corresponding version information from the obtained environment dependencies are extracted sequentially, and corresponding installation instructions are configured for each dependency. The service file's execution instructions are combined with the above instructions to form an instruction set, which is then filled into a template conforming to the image script syntax to automatically generate the service's script file. Finally, the script file is used to build the service image and deploy it to the specified container.

[0091] Step S102: The container information proxy module receives the request from the algorithm model and data transformation method deployment module. It creates an address table recording the requesting IP address, received port, and parameter configuration information, and adds container status information, which is updated periodically by the container management tool. It concatenates the IP address and corresponding port recorded in the address table to obtain the "container address" attribute. It determines the container status and assigns a value to the "enabled" attribute. It obtains a list containing the attributes of number, name, type, parameters, whether enabled, and container address, and forwards this list to the component storage module.

[0092] In step S102, the container management tool periodically obtains container status information and assigns a value to the "Enabled" attribute based on the container status information. For example... Figure 3 As shown, the specific steps are as follows:

[0093] Step S102-1: The status is "created", and the "Enable" attribute is set to "0".

[0094] Step S102-2: The status is "running", and the "Enable" attribute is set to "1".

[0095] Step S102-3: The status is "paused", and the "Enable" attribute is set to "0".

[0096] Step S102-4: The status is "exited", and the "Enable" attribute is set to "0".

[0097] Step S102-5: The status is "dead" and the "Enable" attribute is set to "0".

[0098] Step S102-6: The status is "healthy", and the "Enable" attribute is set to "1".

[0099] Step S102-7: The status is "unhealthy", and the "Enable" attribute is set to "0".

[0100] Step S103: The component storage module receives a list containing the algorithm capsule and data transformation method's ID, name, type, parameters, whether it is enabled, and container address attributes. Simultaneously, the component storage module allocates storage space for each attribute field in the obtained list and performs a "merge" operation on the received list content. This operation compares the attribute fields in the received list and the storage space, copies identical or similar fields, and fills other fields with default values. The merged result is then categorized into algorithm capsule components and data transformation components based on the "type" attribute.

[0101] Step S104: The component visualization module accesses the component storage module to obtain information about all components, visualizes the component icons in the component properties using the graph editing engine, and supports component grouping, collapsing, searching, dragging and dropping capabilities.

[0102] Step S105: Arrange the algorithm capsule components and data transformation components through the component orchestration module, connect all components required for the algorithm pipeline, configure component parameters, and thus complete the construction of the algorithm pipeline model. Request the template generation module and pass the name, type, version, and component information included in the algorithm pipeline model.

[0103] In step S105, constructing the algorithm pipeline model includes the following steps:

[0104] Step S105-1: Generate the visual connection endpoints of the component based on the number of parameters in the component constraint attributes. The upper endpoint of the component is used as the input endpoint, and the lower endpoint is used as the output endpoint.

[0105] Step S105-2: Connect the input and output ends of each visualization component according to the algorithm pipeline flow, and configure the parameters of each component. The configuration of different component parameters changes automatically according to the dependency relationship between parameters, such as linear relationship or conditional relationship between different parameters.

[0106] Step S106: The template generation module processes the received component information and extracts the component number sequence and component constraint sequence based on the component information contained in the algorithm pipeline model. The resulting component number sequence, component constraint sequence, name, type, version, etc., are combined into an attribute list and passed to the component storage module.

[0107] In step S106, the method for processing the component information contained in the algorithm pipeline model includes the following steps:

[0108] Step S106-1: Convert the component information into a JSON format text object, split the key-value pairs in the text object that match the component configuration information, and store them in an object list consisting of the component number and component parameter attributes. Also split the key-value pairs that match the component connection relationship and store them in an object list consisting of the component number and component connection object attributes.

[0109] Step S106-2: Use depth-first search or breadth-first search on the list of objects containing component connection relationships to extract the component connection order. Add objects containing the component number sequence and component constraint sequence from the common template attributes. According to the obtained component connection order, insert the component numbers from the component attributes as elements into the component number sequence object. Similarly, according to the obtained component connection order, insert the configuration information of each component as elements into the component constraint sequence object.

[0110] Step S106-3: Combine the component number sequence and component constraint sequence with the received name, type, version and other attributes to form a new attribute list.

[0111] Step S107: The component storage module receives the attribute list from the template generation module, allocates storage space according to the attribute fields of the common template, and fills the storage space of the common template with the contents of the attribute list. It then requests and transmits the common template information to the component visualization module to enable the visual drag-and-drop functionality of the common template.

[0112] Example 2:

[0113] The purpose of this embodiment is to provide an algorithm pipeline model optimization system for software systems.

[0114] An algorithm pipeline model optimization system for software systems is applied to optimize the algorithm pipeline model constructed by the aforementioned algorithm pipeline model construction system for software systems, including:

[0115] The algorithm pipeline parsing module is used to parse the algorithm pipeline model to be optimized and obtain the task execution sequence of the algorithm pipeline model based on the parsing results;

[0116] The algorithm pipeline execution module is used to calculate the weighted sum of evaluation metrics of the current algorithm pipeline model based on the obtained task execution sequence of the algorithm pipeline model; and to optimize the parameters of the algorithm pipeline model according to the difference between the weighted sum of evaluation metrics and the expected threshold, so as to obtain the optimized algorithm pipeline model; wherein, in the optimization process, different versions of the algorithm pipeline model are obtained and stored.

[0117] The model information display and version switching module is used to display different versions of the algorithm pipeline model and automatically configure the parameters of the corresponding version of the algorithm pipeline model based on the selected version, so as to realize the version switching of the algorithm pipeline model.

[0118] In specific implementation, the optimization of the parameters of the algorithm pipeline model is as follows: based on the changing trend of the results during the optimization process, the parameters of the algorithm pipeline model are continuously corrected using the bisection method.

[0119] In practice, the evaluation metrics include, but are not limited to, the accuracy of classification tasks, the mean squared error of regression tasks, and the precision of recommendation tasks.

[0120] Specifically, for ease of understanding, the following detailed description of the solution in this embodiment is provided in conjunction with the accompanying drawings:

[0121] An algorithm pipeline model optimization system for software systems, comprising:

[0122] The algorithm pipeline parsing module is responsible for converting the incoming algorithm pipeline model information into a JSON-formatted text object. It reads the text object and breaks it down into component connection relationships and component configurations. Using depth-first search or breadth-first search, it extracts the connection order of the components, further obtaining the algorithm pipeline model attribute list, and storing and maintaining each attribute of the algorithm pipeline model. Simultaneously, the algorithm pipeline parsing module fills some attributes of the algorithm pipeline model into the task execution sequence, further obtaining the task execution sequences corresponding to different algorithm pipeline models. The algorithm pipeline parsing module communicates with the algorithm pipeline execution module and transmits task execution sequence information.

[0123] The algorithm pipeline execution module is responsible for receiving task execution sequence information. It sends requests sequentially according to the container address sequence in the task execution sequence, and calculates the final return value according to the rules of the evaluation index sequence. Each evaluation index in the sequence is added according to its weight to obtain a weighted sum, which serves as the sole criterion for evaluating the algorithm pipeline model. The evaluation index sequence is then submitted to the algorithm pipeline parsing module for storage.

[0124] The model information display and version switching module, working in conjunction with the component orchestration device, displays the evaluation metric sequence and parameter configurations of all versions of the algorithm pipeline model, and arranges the different versions of the algorithm pipeline model in descending order according to their adaptability to different application scenarios. Version switching of the algorithm pipeline model is achieved by automatically filling in the parameter configurations of different versions during orchestration.

[0125] like Figure 2 As shown, the optimization system, in practical application, specifically performs the following process:

[0126] Step S201: The component orchestration module orchestrates the algorithm pipeline model and submits it to the algorithm pipeline parsing module, while defining the weight ratio and expected threshold of each evaluation indicator. The expected threshold is the value that the optimization result is expected to achieve.

[0127] Step S202: The algorithm pipeline parsing module analyzes the JSON description of the algorithm pipeline model using JSON processing technology to obtain the attribute list of the algorithm pipeline model. Storage space is allocated for this list, and the contents of the attribute list are filled into the storage space. Simultaneously, based on some attributes in the attribute list, the module requests the container address sequence of the task execution sequence from the component storage module, ultimately obtaining the task execution sequence of the algorithm pipeline model.

[0128] Specifically, in step S202, such as Figure 5 As shown, the steps to obtain the algorithm pipeline model and the task execution sequence attribute list are as follows:

[0129] Step S202-1: The algorithm pipeline parsing module converts the received list of algorithm pipeline models into a JSON-formatted text object.

[0130] Step S202-2: Split the key-value pairs in the text object that match the component configuration information and store them in an object list consisting of the component number and component parameter attributes. Split the key-value pairs that match the component connection relationship and store them in an object list consisting of the component number and component connection object attributes.

[0131] Step S202-3: Use depth-first search or breadth-first search to extract the component connection order from the list of objects containing component connection relationships.

[0132] Step S202-4: Add the algorithm pipeline model object. According to the connection order of the obtained components, insert the component numbers in the component attributes into the component number sequence attribute of the algorithm pipeline model object as elements. Similarly, according to the connection order of the obtained components, insert the configuration information of each component into the component constraint sequence attribute of the algorithm pipeline model object as elements.

[0133] Step S202-5: Obtain an attribute list consisting of number, name, version, component number sequence and component constraint sequence, allocate storage space for the list, and fill the storage space with the contents of the attribute list.

[0134] Step S202-6: Following the order of components in the component number sequence, sequentially request the component storage module to retrieve the container addresses of the corresponding components, ultimately forming the container address sequence of the task execution sequence. Use the parameters in the component constraint sequence as the parameter sequence in the task execution sequence to obtain the task execution sequence of the algorithm pipeline model.

[0135] Step S203: The algorithm pipeline parsing module obtains the task execution sequence of the algorithm pipeline model. This task execution sequence information is input into the algorithm pipeline execution module to obtain a weighted sum of evaluation metrics. Then, based on the difference between the weighted sum of evaluation metrics and the expected threshold, the algorithm pipeline model parameters are continuously optimized, ultimately resulting in the optimized algorithm pipeline model. During the optimization process, a series of algorithm pipeline model versions are generated. These optimized algorithm pipeline model versions are tracked and recorded.

[0136] In step S203, the specific steps for the algorithm pipeline execution module to obtain the weighted sum of evaluation metrics are as follows: Figure 6 As shown: Following the RESTful interface design style, the contents of the container address sequence in the task execution sequence are sequentially converted into URL paths in string format. Simultaneously, the contents of the parameter sequence in the task execution sequence are sequentially stored in a hash table, and the hash table key is inserted into the URL path string as the parameter name. This results in a list of URL paths with request parameters and a hash table corresponding to each URL, which are saved as an API list. The URL paths in the list are requested sequentially, with the corresponding hash table as the parameter. Before each request, the routing table (which records request history and return results) is accessed. If a request address with the same parameters exists in the routing table, the corresponding result is returned directly. If not, the request continues, and the request address, parameter content, and return result are stored in the routing table. The return value of each request is used as the input for the next request, and this process is repeated. The final return value is calculated according to the rules of the evaluation indicator sequence, and the weighted sum of each evaluation indicator in the sequence is obtained by adding them according to their weight proportions.

[0137] The specific steps for optimizing the algorithm pipeline model are as follows:

[0138] Step S203-1: Obtain the weighted sum of evaluation metrics for the algorithm pipeline model according to the calculation formula, compare it with the expected threshold, and record the difference between the two. The formula for obtaining the weighted sum of evaluation metrics is as follows:

[0139]

[0140] In the formula, Y is the weighted sum of evaluation indicators; f i (g) represents the evaluation metrics of the algorithm pipeline model; g represents the algorithm pipeline model, and different versions of the algorithm pipeline model are denoted as g. i (x1,x2,…,x n ), where i represents the version of the algorithm pipeline model, x i Indicates a parameter variable; n i For the weights of different evaluation indicators, n i Satisfying n1+n2+…+n n =1.

[0141] Step S203-2: Add a new version of the algorithm pipeline model, set the parameters to the average value of the specified parameter range [a, b], and when changing the parameters, the parameter changes should follow the following formula:

[0142]

[0143] X is the set of parameter variables; A ij The calculation method for the parameter variables in version i; T(y) i ) represents the changing trend of the weighted sum difference between the expected threshold and the evaluation index in version i; rule(x) indicates that there are corresponding rules for changing different parameter variables, and rule(x)→A means that these rules are used as conditions to affect the adjustment results of parameter variables.

[0144] Step S203-3: Calculate the weighted sum of evaluation metrics for the newly added algorithm pipeline model. Compare the weighted sum of evaluation metrics with the expected threshold to obtain a new difference. If the difference narrows and the changed parameter is greater than the previous version parameter, then set 'a' in [a, b] to the previous version parameter, and leave 'b' unchanged. If the difference narrows and the changed parameter is less than the previous version parameter, then set 'b' in [a, b] to the previous version parameter, and leave 'a' unchanged. If the difference increases and the changed parameter is greater than the previous version parameter, then set 'b' in [a, b] to the previous version parameter, and leave 'a' unchanged. If the difference increases and the changed parameter is less than the previous version parameter, then set 'a' in [a, b] to the previous version parameter, and leave 'b' unchanged.

[0145] Step S203-4: Repeat steps S203-2 and S203-3 until the weighted sum of the evaluation indicators no longer changes significantly, or the expected threshold has been reached.

[0146] Step S204, the model information display and version switching module, defines the weight ratio of evaluation indicators under different application scenarios. Based on the calculation results of all versions under different evaluation indicator weight ratios, the adaptation priority of different versions of the algorithm pipeline model under different application scenarios is obtained and displayed in descending order. Different parameter configurations of the algorithm pipeline model are automatically filled in to achieve version switching of the algorithm pipeline model.

[0147] In step S204, the adaptation priority of different versions of the algorithm pipeline model under different application scenarios is obtained. The specific steps to switch the algorithm pipeline model version are as follows:

[0148] Step S204-1: Calculate the weighted sum of the evaluation indicators for all recorded models based on the weight percentages of the evaluation indicators under different application scenarios.

[0149] Step S204-2: Sort all versions of the algorithm pipeline model according to the weighted sum of evaluation indicators, and display them in descending order in the component orchestration module in the form of a table.

[0150] Step S204-3: Configure the functional attributes of the table. When selecting a certain version of the algorithm pipeline model in the table, record the parameter configuration information and fill the parameter configuration information into the parameter configuration panel of each component.

[0151] Example 3:

[0152] The purpose of this embodiment is to provide a method for constructing an algorithm pipeline model for a software system.

[0153] A method for constructing an algorithm pipeline model for a software system, based on the algorithm pipeline model construction system for a software system described in Embodiment 1, the method comprising:

[0154] The module is based on the algorithm model and data transformation method deployment to build and deploy containers corresponding to the algorithm model and data transformation method.

[0155] The container information proxy module converts container information into component properties and forwards them to the component storage module; and, based on the container's state, it requests in real time whether the component in the component storage module has enabled property modification.

[0156] The component storage module stores container components and maintains component attribute information based on requests from the container information proxy module; it also stores algorithm pipeline models, task execution sequences, and common templates.

[0157] Based on the component orchestration module, the components in the component storage module are orchestrated using a graphical drag-and-drop interface to build the algorithm pipeline model.

[0158] Furthermore, the implementation process of the method described in this embodiment has been described in detail in the system described in Embodiment 1, so it will not be repeated here.

[0159] Example 4:

[0160] The purpose of this embodiment is to provide a method for optimizing algorithm pipeline models in software systems.

[0161] An algorithm pipeline model optimization method for software systems includes:

[0162] The algorithm pipeline model to be optimized is analyzed, and the task execution sequence of the algorithm pipeline model is obtained based on the analysis results;

[0163] Based on the obtained task execution sequence of the algorithm pipeline model, the weighted sum of the evaluation metrics of the current algorithm pipeline model is calculated; and the parameters of the algorithm pipeline model are optimized according to the difference between the weighted sum of the evaluation metrics and the expected threshold to obtain the optimized algorithm pipeline model; wherein, in the optimization process, different versions of the algorithm pipeline model are obtained and stored.

[0164] Different versions of the algorithm pipeline model are displayed, and the parameters of the corresponding version of the algorithm pipeline model are automatically configured based on the selected version, so as to realize the version switching of the algorithm pipeline model.

[0165] Furthermore, the system described in this embodiment corresponds to the system described in Embodiment 2, and its technical details have been described in detail in Embodiment 2, so they will not be repeated here.

[0166] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0167] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A system for constructing algorithm pipeline models for software systems, characterized in that, include: The algorithm model and data conversion method deployment module is used to receive the algorithm model, data conversion method, environment dependency information, and parameter information to be deployed, and to build and deploy the container corresponding to the algorithm model and data conversion method; wherein, the algorithm model and data conversion method are represented in the form of container components; the deployment of the algorithm model and data conversion method specifically involves: encapsulating the executable files of the received algorithm model and data conversion method to be deployed into service files and copying them to a specified working directory, and automatically creating and deploying the container image of the algorithm model and data conversion method; The container information proxy module is used to convert container information from the algorithm model and data transformation method deployment module into component properties and forward them to the component storage module; and, based on the container's status, to request in real time whether the component in the component storage module has enabled property modification. The component storage module is used to store container components and maintain component attribute information based on requests from the container information proxy module; as well as to store algorithm pipeline models, task execution sequences, and common templates. The component orchestration module is used to orchestrate components in the component storage module using a graphical drag-and-drop interface to build an algorithm pipeline model. This includes: generating connection endpoints for the visualized components based on the number of parameters in the component constraint attributes, with the upper endpoint of the component serving as the input endpoint and the lower endpoint as the output endpoint; connecting the input and output endpoints of each visualized component according to the algorithm pipeline flow; configuring the parameters of each component; and automatically changing the parameter configurations of different components according to the dependencies between parameters.

2. The algorithm pipeline model construction system for software systems as described in claim 1, characterized in that, The system also includes a component visualization module and a template generation module. The visualization module is used to visualize the components and common templates in the component storage module. The template generation unit is used to generate corresponding common templates and store them in the component storage module based on the algorithm pipeline model obtained from the component orchestration results through serialization processing.

3. The algorithm pipeline model construction system for software systems as described in claim 1, characterized in that, The algorithm model is deployed as an algorithm capsule component in the form of an algorithm capsule. The algorithm capsule includes external attributes and internal structure. The external attributes include the trained algorithm model source code and the external dependencies of the algorithm capsule. The internal structure includes an input / output interface layer, a core layer, and an environment dependency layer built based on the external attributes. The core layer is used to preprocess the input data of the current algorithm model's corresponding task according to its corresponding application domain; Based on the preprocessed input data, request the data processing service of the algorithm model to obtain the processing result.

4. The algorithm pipeline model construction system for software systems as described in claim 1, characterized in that, The task execution sequence specifically refers to the execution order of each processing step in the algorithm pipeline model.

5. An algorithm pipeline model optimization system for software systems, characterized in that, Its application is to the optimization of an algorithm pipeline model constructed based on an algorithm pipeline model construction system for software systems as described in any one of claims 1-4, including: The algorithm pipeline parsing module is used to parse the algorithm pipeline model to be optimized and obtain the task execution sequence of the algorithm pipeline model based on the parsing results; The algorithm pipeline execution module is used to calculate the weighted sum of evaluation metrics of the current algorithm pipeline model based on the obtained task execution sequence of the algorithm pipeline model; and to optimize the parameters of the algorithm pipeline model according to the difference between the weighted sum of evaluation metrics and the expected threshold, so as to obtain the optimized algorithm pipeline model; wherein, in the optimization process, different versions of the algorithm pipeline model are obtained and stored. The model information display and version switching module is used to display different versions of the algorithm pipeline model and automatically configure the parameters of the corresponding version of the algorithm pipeline model based on the selected version, so as to realize the version switching of the algorithm pipeline model.

6. The algorithm pipeline model optimization system for software systems as described in claim 5, characterized in that, The optimization of the parameters of the algorithm pipeline model specifically involves using a bisection method to continuously correct the parameters of the algorithm pipeline model based on the changing trend of the results during the optimization process.

7. The algorithm pipeline model optimization system for software systems as described in claim 5, characterized in that, The evaluation metrics include, but are not limited to, the accuracy of classification tasks, the mean squared error of regression tasks, and the precision of recommendation tasks.

8. A method for constructing an algorithm pipeline model for a software system, characterized in that, It is based on an algorithm pipeline model construction system for software systems as described in any one of claims 1-4, the method comprising: The module is based on the algorithm model and data transformation method deployment to build and deploy containers corresponding to the algorithm model and data transformation method. The container information proxy module converts container information into component properties and forwards them to the component storage module; and, based on the container's state, it requests in real time whether the component in the component storage module has enabled property modification. The component storage module stores container components and maintains component attribute information based on requests from the container information proxy module; it also stores algorithm pipeline models, task execution sequences, and common templates. Based on the component orchestration module, the components in the component storage module are orchestrated using a graphical drag-and-drop interface to build the algorithm pipeline model.

9. A method for optimizing an algorithm pipeline model in a software system, characterized in that, It is based on an algorithm pipeline model optimization system for software systems as described in any one of claims 1-4, comprising: The algorithm pipeline model to be optimized is analyzed, and the task execution sequence of the algorithm pipeline model is obtained based on the analysis results; Based on the obtained task execution sequence of the algorithm pipeline model, the weighted sum of the evaluation metrics of the current algorithm pipeline model is calculated; and the parameters of the algorithm pipeline model are optimized according to the difference between the weighted sum of the evaluation metrics and the expected threshold to obtain the optimized algorithm pipeline model; wherein, in the optimization process, different versions of the algorithm pipeline model are obtained and stored. Different versions of the algorithm pipeline model are displayed, and the parameters of the corresponding version of the algorithm pipeline model are automatically configured based on the selected version, so as to realize the version switching of the algorithm pipeline model.

Images