Algorithm service construction method and device, equipment and medium
By deploying algorithm services in a distributed computing cluster, abstracting algorithm interfaces and setting scheduling logic, the computing bottlenecks and high implementation costs of distributed computing and algorithm service construction are solved, and efficient algorithm service integration and operation and maintenance are achieved.
Patent Information
- Application Number
- CN202411423871.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, the construction of distributed computing and algorithm services has computing bottlenecks and high implementation costs, and lacks the ability to directly integrate with algorithm services, resulting in high development difficulty and high maintenance costs.
By deploying at least two computing node environments, installing a distributed computing cluster, and abstracting different algorithms into a unified abstract algorithm interface, setting up scheduling logic and external interfaces, integrating algorithm services, and deploying them in a distributed computing cluster.
It realizes the integration of distributed algorithm services in distributed services, provides a unified interface for users to use, supports algorithm teams to expand algorithm capabilities more flexibly, and improves the operation and maintenance capabilities of algorithms.
Smart Images
Figure CN119449564A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of distributed computing, and in particular to an algorithm service construction method, device, equipment and medium. Background Art
[0002] With the continuous development of IT technology, AI intelligent technology is increasingly being used, and many companies have applied machine learning, deep learning and other technologies in commercialization. However, in the existing technology, there are some significant technical difficulties in the construction of distributed computing and algorithm services. The traditional single-node computing model is often difficult to meet the needs of efficient processing when facing large-scale data processing. With the rapid growth of data scale, the computing resources of a single node (such as CPU, GPU, memory, etc.) are often insufficient to support the real-time operation of complex algorithms, resulting in computing bottlenecks and performance degradation. In order to solve this problem, many systems have begun to adopt a distributed computing architecture to improve computing power by distributing computing tasks to multiple nodes. However, building and managing distributed computing systems requires developers to have a fairly high technical threshold, involving many complex issues such as inter-node communication, task scheduling, load balancing, fault tolerance, etc., resulting in high implementation costs.
[0003] Although the existing distributed computing frameworks and tools are rich, they often lack the ability to directly integrate with algorithm services. Developers usually need to design and implement mechanisms to embed algorithms into distributed computing frameworks, which increases system complexity and development difficulty. In addition, how to efficiently provide a Web interface for algorithm services is also an urgent problem to be solved. When encapsulating algorithm services as Web interfaces, current technical solutions usually require developers to design and develop interface logic by themselves. There is a lack of unified specifications and convenient tool support, resulting in low development efficiency and high maintenance costs. Summary of the invention
[0004] The embodiments of the present application provide an algorithm service construction method, apparatus, device and medium to support the expansion of computing power by utilizing the computing power of multiple nodes.
[0005] According to one aspect of the present application, a method for constructing an algorithm service is provided, the method comprising:
[0006] Deploy at least two computing node environments, and install distributed computing clusters in at least two computing node environments respectively;
[0007] For different businesses, different algorithms are abstracted into a unified abstract algorithm interface, and the algorithm scheduling logic is set to respond to the computing tasks and parallelism specified by the user;
[0008] Determine the external interface provided by the algorithm service to determine the data source, algorithm logic and output method through the external interface;
[0009] The abstract algorithm interface, the scheduling logic and the external interface are integrated to form an algorithm service, which is deployed in the distributed computing cluster.
[0010] According to one aspect of the present application, there is provided an algorithm service construction device, the device comprising:
[0011] A deployment module, used to deploy at least two computing node environments and install a distributed computing cluster in the at least two computing node environments respectively;
[0012] The algorithm abstraction module is used to abstract different algorithms into a unified abstract algorithm interface for different businesses, and set the algorithm scheduling logic to respond to the computing tasks and parallelism specified by the user;
[0013] An external interface determination module is used to determine the external interface provided by the algorithm service to the outside world, so as to determine the data source, algorithm logic and output mode through the external interface;
[0014] The algorithm service integration module is used to integrate the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, which is deployed in the distributed computing cluster.
[0015] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:
[0016] at least one processor; and
[0017] A memory connected to at least one processor for data processing; wherein,
[0018] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the algorithm service construction method of any embodiment of the present application.
[0019] According to another aspect of the present application, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to implement the algorithm service construction method of any embodiment of the present application when executed by a processor.
[0020] The technical solution of the embodiment of the present application deploys at least two computing node environments, and installs a distributed computing cluster in at least two computing node environments respectively; for different businesses, different algorithms are abstracted into a unified abstract algorithm interface, and the scheduling logic of the algorithm is set to respond to the computing tasks and parallelism specified by the user; the external interface provided by the algorithm service is determined to determine the data source, algorithm logic and output method through the external interface; the abstract algorithm interface, the scheduling logic and the external interface are integrated to form an algorithm service, which is deployed in the distributed computing cluster. The above scheme can be used for distributed services, and can integrate distributed algorithm services to provide a unified interface for users to use, support the algorithm team to expand the algorithm capabilities more flexibly, and improve the algorithm's operation and maintenance capabilities.
[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0023] Figure 1 A flowchart of an algorithm service construction method provided in an embodiment of the present application;
[0024] Figure 2 The overall hierarchical structure diagram of the algorithm construction service provided in the embodiment of the present application;
[0025] Figure 3 A flowchart of an algorithm service construction method provided in another embodiment of the present application;
[0026] Figure 4 A flowchart of a computing task processing provided by another embodiment of the present application;
[0027] Figure 5 A schematic diagram of the structure of an algorithm service construction device provided in an embodiment of the present application;
[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0030] It should be noted that the terms "first", "second", "third", "fourth", "actual", "preset", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0031] Figure 1 The flowchart of an algorithm service construction method provided in the embodiment of the present application is applicable to the case where an algorithm service of a distributed computing service is constructed. The method can be executed by an algorithm service construction device, which can be implemented in the form of hardware and / or software, and the algorithm service construction device can be configured in an electronic device. Figure 1 As shown, the method includes:
[0032] S110: Deploy at least two computing node environments, and install a distributed computing cluster in the at least two computing node environments respectively.
[0033] The computing node environment may be a physical environment that can support computing, such as a virtual machine or a physical machine node, and an operating environment may be installed, such as a python (3.10 or above) environment. The distributed computer cluster is a cluster that supports distributed computing, such as a ray cluster.
[0034] Exemplarily, an environment for supporting code execution may be first constructed, at least two computing node environments may be deployed in an electronic device, and distributed computing clusters may be installed in the at least two computing node environments to support distributed computing.
[0035] S120 . For different businesses, different algorithms are abstracted into a unified abstract algorithm interface, and the scheduling logic of the algorithm is set to respond to the computing tasks and parallelism specified by the user.
[0036] In order to provide a unified interface form to the outside world in the embodiments of the present application, an abstract interface is designed to standardize the behavior of the algorithm logic. Different algorithms implement specific logic according to different business needs. For example, for the Python language, all algorithm logics must be implemented: first process the data, then perform algorithm prediction or detection on the processed data after processing, and return the calculation results after calculation. The above process is an idea of algorithm abstraction, which is an abstraction based on the capabilities that the algorithms currently on the market need to provide. Based on this, it is necessary to define data processing functions, algorithm prediction or detection functions, such as model training, in the abstract algorithm interface to realize the abstraction of different algorithms.
[0037] In the embodiment of the present application, the scheduling logic of the algorithm needs to be set, that is, the computing task can be split and allocated to the distributed computer group for distributed computing in response to the computing task and parallelism specified by the user. For example, if the computing task input by the user is received through the interface and the parallelism is 6, the computing task needs to be split into 6 subtasks, which are run in parallel on the nodes of the distributed cluster for parallel execution.
[0038] S130. Determine the external interface provided by the algorithm service to determine the data source, algorithm logic and output method through the external interface.
[0039] For example, in order to clarify the data source, algorithm logic and output method, it is also necessary to determine the external interface provided by the algorithm service to determine the data source algorithm logic and output method through the external interface. For example, source defines where the data source of the algorithm comes from, which can come from API, database, Kafka, etc., which can be expanded by itself. Algorithm is used to define the algorithm logic, which can be anomaly detection, trend prediction, etc., which can be expanded by the algorithm team. Sink is used to define the output method of the calculation results, which can be API, database, Kafka, etc., which can be expanded by itself.
[0040] S140, integrating the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, and deploying it in the distributed computing cluster.
[0041] Exemplarily, the algorithm is defined as an interface and can be deployed in a distributed manner on the nodes of a distributed computing cluster to form multiple service instances and achieve load balancing. Through the interface parameters passed by the caller, the backend builds different algorithm services to run on different distributed computing cluster nodes. After the task is completed, the results are summarized and output.
[0042] For example, the overall hierarchical structure diagram of building an algorithm service is as follows: Figure 2 As shown, the infrastructure includes a physical environment that supports program operation, and a Ray cluster is installed on the infrastructure to provide distributed computing services. The distributed computing cluster can support training, prediction and other algorithms, integrate the FastAPI interfaces of the Ray service, form an algorithm service, and deploy it in the distributed computing cluster for use by the consumer service party.
[0043] The technical solution of the embodiment of the present application deploys at least two computing node environments, and installs a distributed computing cluster in at least two computing node environments respectively; for different businesses, different algorithms are abstracted into a unified abstract algorithm interface, and the scheduling logic of the algorithm is set to respond to the computing tasks and parallelism specified by the user; the external interface provided by the algorithm service is determined to determine the data source, algorithm logic and output method through the external interface; the abstract algorithm interface, the scheduling logic and the external interface are integrated to form an algorithm service, which is deployed in the distributed computing cluster. The above scheme can be used for distributed services, and can integrate distributed algorithm services to provide a unified interface for users to use, support the algorithm team to expand the algorithm capabilities more flexibly, and improve the algorithm's operation and maintenance capabilities.
[0044] Figure 3 This is a flowchart of an algorithm service construction method provided by another embodiment of the present application. The present embodiment is optimized based on the above embodiment. For solutions not described in detail in the present embodiment, please refer to the above embodiment. Figure 3 As shown, the method of the embodiment of the present application specifically includes the following steps:
[0045] S210: Deploy at least two computing node environments, and install a distributed computing cluster in the at least two computing node environments respectively.
[0046] S220. For different businesses, different algorithms are abstracted into a unified abstract algorithm interface.
[0047] S230, setting task splitting logic to split the computing tasks and parallelism specified by the user through the task splitting logic, and assigning the subtasks obtained by the splitting to each computing node environment for processing.
[0048] Exemplarily, logic for splitting tasks can be set. When the parallelism of a computing task is received from a user, the computing task can be split based on the parallelism to obtain subtasks, and the split subtasks can be assigned to each computing node environment for processing to achieve parallel processing of the computing task.
[0049] S240: Define a data source interface to obtain a data source set by a user through the data source interface.
[0050] For example, a data source interface can be defined so that the user can input the data source through this interface, thereby clarifying the data source and obtaining the data to be processed according to the data source. For example, if the "type" is defined as "Metric" in the source, it can be determined that the original data is obtained from the Metric system.
[0051] S250, defining an algorithm logic interface, so as to obtain the algorithm logic set by the user through the algorithm logic interface, and determining a corresponding abstract algorithm interface.
[0052] For example, an algorithm logic interface can be defined so that the user can input specific algorithm logic through the algorithm logic interface, clarify which specific algorithm logic needs to be executed, and then call the corresponding abstract algorithm interface. For example, if the "type" is defined as "Dtmos" in algorithm, the algorithm logic can be determined to be the Dtmos anomaly detection algorithm.
[0053] S260: Define an output mode interface to obtain the calculation result output mode set by the user through the output mode interface.
[0054] For example, an output mode interface can be defined so that the user can set the output mode of the calculation result through the output mode interface, so that the calculation result is output in the output mode set by the user. For example, if the "type" is defined as "kafka" in the sink, it can be determined that the calculation result is output to the kafka message component.
[0055] S270, integrating the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, and connecting with the front-end page to obtain the interface parameters transmitted by the user.
[0056] Exemplarily, the abstract algorithm interface, scheduling logic and external interface obtained above can be formed into an algorithm service to be directly provided to users for use. Users can directly process computing tasks by inputting parameters through the interface in the algorithm service.
[0057] In the embodiments of the present application, Ray and FastAPI open source tools are used to help build web services, and interfaces can be provided to callers. Algorithm services are implemented through the input and output parameters of the interface in the above scheme and subsequent algorithm scheduling. Algorithm services are general and can be integrated without being specific to a certain algorithm. As long as the business needs it, as long as the algorithm logic is generated in the above manner to implement the abstract interface of the algorithm, it is integrated into the algorithm service and can be provided to the outside in the form of an interface. The parameters of the interface describe the source of the original data, the parameters for running the algorithm, and the way the results are output after the algorithm is run.
[0058] S280. Deploy the algorithm service in each distributed computing cluster.
[0059] The embodiment of the present application provides a method for constructing an algorithm service, deploying at least two computing node environments, and installing distributed computing clusters in at least two computing node environments respectively. For different businesses, different algorithms are abstracted into a unified abstract algorithm interface. Task splitting logic is set to split the computing tasks and parallelism specified by the user through the task splitting logic, and the subtasks obtained by the split are assigned to each computing node environment for processing. A data source interface is defined to obtain the data source set by the user through the data source interface. An algorithm logic interface is defined to obtain the algorithm logic set by the user through the algorithm logic interface, and the corresponding abstract algorithm interface is determined. An output mode interface is defined to obtain the calculation result output mode set by the user through the output mode interface. The abstract algorithm interface, the scheduling logic and the external interface are integrated to form an algorithm service, and connected to the front-end page to obtain the interface parameters passed by the user. The algorithm service is deployed in each distributed computing cluster. The above scheme can generate algorithm logic to implement the abstract interface of the algorithm, integrate it into the algorithm service, and provide it to the outside in the form of an interface, which supports the algorithm team to expand the algorithm capability more flexibly and improves the operation and maintenance capability of the algorithm.
[0060] In an embodiment of the present application, the method further includes:
[0061] Obtaining interface parameters passed by the user through the front-end page, and running the algorithm service in each distributed computing cluster according to the interface parameters;
[0062] The calculation results of the algorithm services in each distributed computing cluster are summarized and output.
[0063] In the embodiment of the present application, before aggregating and outputting the calculation results of the algorithm services in each distributed computing cluster, the method further includes:
[0064] Obtaining a computing task and a degree of parallelism specified by a user, and splitting the computing task according to the degree of parallelism to obtain subtasks;
[0065] Assign subtasks to each computing node environment for processing.
[0066] Exemplarily, in the actual process of processing computing tasks, the user inputs interface parameters through the front-end page, and runs the algorithm service in the distributed computing cluster according to the interface parameters to realize distributed computing of computing tasks, obtain multi-way computing results in the process of multi-way parallel computing, and summarize the multi-way computing results. For example, it is necessary to identify 15 days of images to find the preset target. If the amount of image data for 15 days is relatively large, the calculation on a single server will be very slow. At this time, we can distribute it to different servers for parallel computing based on the 15 days of images, which speeds up the speed. The specific splitting of 6 subtasks (subtasks run independently) can be controlled according to the incoming parameter parallelism (for example, 6). The splitting can be evenly split according to the amount of data, or it can be split according to the remaining resources of the distributed computer group. After the split is completed on different subtasks, each subtask is aggregated and summarized. For example Figure 4 As shown in the figure, the external caller makes an interface call through the web interface layer, and uses the task scheduling layer to allocate the subtasks split according to the concurrency. Assuming the concurrency is 6, 6 subtasks are obtained and allocated to 3 Ray nodes. Each Ray node processes 2 parallel subtask pipeline acts, and finally summarizes the processing results of each Ray node and outputs them.
[0067] In the embodiment of the present application, the interface parameters passed by the user are obtained through the front-end page, and the algorithm service in each distributed computing cluster is run according to the interface parameters, including:
[0068] Obtain the interface parameters passed by the user through the front-end page and assign them to the corresponding external interface;
[0069] The corresponding abstract algorithm interface is determined through the interface parameters in the external interface, and the abstract algorithm interface is run to execute the corresponding algorithm logic.
[0070] Exemplarily, the interface parameters passed by the user obtained from the front-end page need to be passed to the external interface so that the external interface can determine the meaning of the parameters. The corresponding abstract algorithm interface is determined by the interface parameters in the external interface, and the algorithm logic corresponding to the abstract algorithm interface is run. For example, when the "type" in the source is "Metric", the algorithm service will go to the data acquisition logic of the abstract algorithm interface MetricSource according to the judgment. When the "type" in the algorithm is "Dtmos", the algorithm service will go to the abstract algorithm interface DtmosTrainer algorithm according to the judgment for calculation. When the "type" in the sink is "kafka", the algorithm service will go to the abstract algorithm interface KafkaSink according to the judgment for result output.
[0071] Figure 5 This is a schematic diagram of the structure of an algorithm service construction device provided in an embodiment of the present application. The device can execute the algorithm service construction method provided in any embodiment of the present application and has the corresponding functional modules and beneficial effects of the execution method. Figure 5 As shown, the device comprises:
[0072] A deployment module 310 is used to deploy at least two computing node environments and install a distributed computing cluster in the at least two computing node environments respectively;
[0073] The algorithm abstraction module 320 is used to abstract different algorithms into a unified abstract algorithm interface for different services, and set the algorithm scheduling logic to respond to the computing tasks and parallelism specified by the user;
[0074] The external interface determination module 330 is used to determine the external interface provided by the algorithm service to determine the data source, algorithm logic and output mode through the external interface;
[0075] The algorithm service integration module 340 is used to integrate the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, which is deployed in the distributed computing cluster.
[0076] In the embodiment of the present application, the algorithm abstraction module 320 sets the scheduling logic of the algorithm in response to the computing task and parallelism specified by the user, including:
[0077] Set the task splitting logic to split the computing tasks and parallelism specified by the user through the task splitting logic, and assign the resulting subtasks to each computing node environment for processing.
[0078] In the embodiment of the present application, the external interface determination module 330 determines the external interface provided by the algorithm service to the outside, so as to determine the data source, algorithm logic and output mode through the external interface, including:
[0079] Define a data source interface to obtain the data source set by the user through the data source interface;
[0080] Define an algorithm logic interface to obtain the algorithm logic set by the user through the algorithm logic interface and determine the corresponding abstract algorithm interface;
[0081] An output mode interface is defined to obtain the calculation result output mode set by the user through the output mode interface.
[0082] In the embodiment of the present application, the algorithm service integration module 340 integrates the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, which is deployed in the distributed computing cluster, including:
[0083] Integrate the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, and connect with the front-end page to obtain the interface parameters passed by the user;
[0084] The algorithm service is deployed in each distributed computing cluster.
[0085] In the embodiment of the present application, the device further includes:
[0086] An operation module is used to obtain the interface parameters passed by the user through the front-end page, and to operate the algorithm service in each distributed computing cluster according to the interface parameters;
[0087] The output module is used to summarize and output the calculation results of the algorithm service in each distributed computing cluster.
[0088] In the embodiment of the present application, the operation module obtains the interface parameters passed by the user through the front-end page, and runs the algorithm service in each distributed computing cluster according to the interface parameters, including:
[0089] Obtain the interface parameters passed by the user through the front-end page and assign them to the corresponding external interface;
[0090] The corresponding abstract algorithm interface is determined through the interface parameters in the external interface, and the abstract algorithm interface is run to execute the corresponding algorithm logic.
[0091] In the embodiment of the present application, before aggregating and outputting the calculation results of the algorithm services in each distributed computing cluster, the device further includes:
[0092] A subtask splitting module is used to obtain a computing task and a degree of parallelism specified by a user, and split the computing task according to the degree of parallelism to obtain subtasks;
[0093] The processing module is used to assign subtasks to each computing node environment for processing.
[0094] An algorithm service construction device provided in an embodiment of the present application can execute an algorithm service construction method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.
[0095] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0096] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the data processing of the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0097] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a data processing unit 19, such as a network card, a modem, a wireless data processing transceiver, etc. The data processing unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0098] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the algorithm service construction method.
[0099] In some embodiments, the algorithm service construction method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the data processing unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the algorithm service construction method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the algorithm service construction method in any other appropriate manner (e.g., by means of firmware).
[0100] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0101] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable algorithm service construction device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0102] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0103] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0104] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data processing (e.g., a data processing network). Examples of data processing networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0105] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a data processing network. The client and server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the information expected by the technical solution of this application can be achieved, and this document is not limited here.
[0107] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A method for constructing an algorithm service, characterized in that: The method comprises: Deploy at least two computing node environments, and install distributed computing clusters in at least two computing node environments respectively; For different businesses, different algorithms are abstracted into a unified abstract algorithm interface, and the algorithm scheduling logic is set to respond to the computing tasks and parallelism specified by the user; Determine the external interface provided by the algorithm service to determine the data source, algorithm logic and output method through the external interface; The abstract algorithm interface, the scheduling logic and the external interface are integrated to form an algorithm service, which is deployed in the distributed computing cluster.
2. The method according to claim 1, characterized in that Set the algorithm's scheduling logic to respond to the user-specified computational tasks and parallelism, including: Set the task splitting logic to split the computing tasks and parallelism specified by the user through the task splitting logic, and assign the resulting subtasks to each computing node environment for processing.
3. The method according to claim 1, characterized in that Determine the external interface provided by the algorithm service to determine the data source, algorithm logic and output method through the external interface, including: Define a data source interface to obtain the data source set by the user through the data source interface; Define an algorithm logic interface to obtain the algorithm logic set by the user through the algorithm logic interface and determine the corresponding abstract algorithm interface; An output mode interface is defined to obtain the calculation result output mode set by the user through the output mode interface.
4. The method according to claim 1, characterized in that: Integrating the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, which is deployed in the distributed computing cluster, includes: Integrate the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, and connect with the front-end page to obtain the interface parameters passed by the user; The algorithm service is deployed in each distributed computing cluster.
5. The method according to claim 1, characterized in that The method further comprises: Obtaining interface parameters passed by the user through the front-end page, and running the algorithm service in each distributed computing cluster according to the interface parameters; The calculation results of the algorithm services in each distributed computing cluster are summarized and output.
6. The method according to claim 5, characterized in that Obtaining interface parameters passed by the user through the front-end page, and running the algorithm service in each distributed computing cluster according to the interface parameters, including: Obtain the interface parameters passed by the user through the front-end page and assign them to the corresponding external interface; The corresponding abstract algorithm interface is determined through the interface parameters in the external interface, and the abstract algorithm interface is run to execute the corresponding algorithm logic.
7. The method according to claim 5, characterized in that Before aggregating and outputting the calculation results of the algorithm services in each distributed computing cluster, the method further includes: Obtaining a computing task and a degree of parallelism specified by a user, and splitting the computing task according to the degree of parallelism to obtain subtasks; Assign subtasks to each computing node environment for processing.
8. An algorithm service construction device, characterized in that: The device comprises: A deployment module, used to deploy at least two computing node environments and install a distributed computing cluster in the at least two computing node environments respectively; The algorithm abstraction module is used to abstract different algorithms into a unified abstract algorithm interface for different businesses, and set the algorithm scheduling logic to respond to the computing tasks and parallelism specified by the user; An external interface determination module is used to determine the external interface provided by the algorithm service to the outside world, so as to determine the data source, algorithm logic and output mode through the external interface; The algorithm service integration module is used to integrate the abstract algorithm interface, the scheduling logic and the external interface to form an algorithm service, which is deployed in the distributed computing cluster.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and A memory connected to the at least one processor for data processing; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the algorithm service construction method described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the algorithm service construction method described in any one of claims 1 to 7 when executed.