Model service method and device, equipment and storage medium

Abstract

The invention discloses a model service method and device, equipment and a storage medium, and relates to the technical field of data processing, and the model service method comprises the steps: obtaining a model service request, and determining the type of the model service request; based on the type, determining a target module for processing the request from a started Modelet model engine; the Modelet model engine is used for performing initial processing on a request based on a target module and performing other processing on the initially processed request based on an original model service externally provided by a started local through a reverse proxy mode, and the Modelet model engine is designed by adopting father and son processes and a reverse proxy architecture and is designed by adopting the father and son processes and the reverse proxy architecture based on the father and son processes and the reverse proxy architecture. After the engine is started, the engine serves as a parent process to start a child process of the original model, so that the Modelet model engine reversely agents the original model, and corresponding original model services are provided. The method guarantees stable operation of large model services.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model data processing, in particular to a model service method and device, equipment and a storage medium. BACKGROUND

[0002] In the AI era, large model services, as the core of AI capabilities, have become the key infrastructure for enterprise-level intelligent applications. Unlike traditional CPU-type services, large model services themselves have problems such as high resource occupation, low reliability guarantee, and low concurrent performance. In addition, in actual enterprise application scenarios, multiple types and multiple inference acceleration frameworks may coexist. Therefore, how to ensure the smooth running of large model services is the basis for service intelligence. SUMMARY

[0003] The main purpose of the present application is to provide a model service method, device and storage medium, which aims to solve the technical problem that it is difficult to ensure the smooth running of large model services in the current technology.

[0004] To achieve the above purpose, the present application provides a model service method applied to a Modelet model engine, which comprises: obtaining a model service request and determining the type of the model service request; determining a target module for processing the model service request from the started Modelet model engine based on the type of the model service request, wherein the Modelet model engine at least includes a model service agent module, a Metric module, a probe module, and a log service module; based on the target module, performing initial processing on the model service request, and based on the original model service provided by the started local through the reverse proxy mode, performing other processing on the model service request after the initial processing, wherein the Modelet model engine is designed based on parent-child processes and a reverse proxy architecture. After the Modelet model engine is started, the Modelet model engine starts the child process of the original model as the parent process, so as to reverse proxy the original model and provide the corresponding original model service.

[0005] In an embodiment, the model service request includes a business request. When the model service request is a business request, the target module includes a model service agent module. The step of performing initial processing on the model service request based on the target module, and performing other processing on the model service request after the initial processing based on the original model service provided by the started local through the reverse proxy mode, comprises: determine a link context corresponding to the service request; based on an authentication component in the model service agent module, load and decrypt authentication information configuration from a preset environment variable configuration to verify the link context; if the verification passes, based on the original model service provided externally by the started local reverse proxy, perform other processing on the initial processed model service request.

[0006] In an embodiment, the target module includes a log service module, and after the step of if the verification passes, based on the original model service provided externally by the started local reverse proxy, performing other processing on the initial processed model service request, the target module includes: if the verification passes, generate first link information corresponding to the verified link; if the verification fails, generate second link information corresponding to the verified link; send the first link information and the second link information to the log service module based on a preset asynchronous mode for the log service module to save and output; wherein the log service module also intercepts and processes traffic output from the original model through parent and child processes of the Modelet model engine and a reverse proxy architecture.

[0007] In an embodiment, the model service request includes an index collection request, and when the model service request is an index collection request, the target module includes a Metric module, and before the step of based on the target module, performing initial processing on the model service request, the target module includes: determine whether a Metric module for index statistics is started on the user side; if the Metric module is started, the steps of based on the Metric module, performing initial processing on the model service request, and based on the original model service provided externally by the started local reverse proxy, performing other processing on the initial processed model service request, include: determine configuration information determined by the user side, based on the configuration information, determine target statistical functions of the Metric module that need to be started; wherein the target statistical functions include Metric interface functions, monitoring and statistical service functions of directly accessing the model itself, and preset customized monitoring and collection related logic functions; based on the started target statistical functions, perform initial processing on the index collection request, and based on the original model service provided externally by the started local reverse proxy, perform other processing on the initial processed index collection request.

[0008] In one embodiment, the model service request includes a health probe request. When the model service request is a health probe request, the target module includes a probe module. The step of performing initial processing on the model service request based on the target module, and performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, includes: Determine the configuration information on the user's end, and determine the health detection method based on the configuration information; Based on the health detection method, a probe mode is determined according to the health detection interface determined by the target module. Based on the detection mode, the model service request is initially processed, and based on the original model service provided externally by the local system through a reverse proxy, the initially processed model service request is further processed.

[0009] In one embodiment, the step of initially processing the model service request based on the detection mode, and then performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, includes: If the detection mode is Direct health detection mode, the model service request is forwarded directly, and the original model service provided to the outside world through the reverse proxy on the local machine is further processed. If the detection mode is inDirect health detection, the probe module performs advanced processing on the model service request. Based on the original model service provided externally via reverse proxy, the advanced processing of the model service request is further processed. The advanced processing includes converting the model service request into a request that meets the requirements of the original model service.

[0010] Furthermore, to achieve the above objectives, this application also proposes a model service apparatus for use in the Modelet model engine, the model service apparatus comprising: The acquisition module is used to acquire model service requests and determine the type of the model service requests; The determination module is used to determine the target module for processing the model service request from the started Modelet model engine based on the type of the model service request, wherein the Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module. The proxy module is used to perform initial processing on the model service request based on the target module, and to perform other processing on the initially processed model service request based on the original model service provided externally through a reverse proxy on the started local machine. The Modelet model engine is designed with a parent-child process and reverse proxy architecture. Based on the parent-child process and reverse proxy architecture, after the Modelet model engine starts, the Modelet model engine starts the child process of the original model as the parent process, so that the Modelet model engine can reverse proxy the original model and provide the corresponding original model service.

[0011] In addition, to achieve the above objectives, this application also proposes a model service device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the model service steps as described above.

[0012] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the model service as described above.

[0013] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the model service as described above.

[0014] One or more technical solutions proposed in this application have at least the following technical effects: Compared to related technologies, in actual enterprise applications, there are often situations where multiple types and multiple inference acceleration frameworks coexist, making it difficult to ensure the stable operation of large model services. This application obtains model service requests and determines the type of the model service request; based on the type of the model service request, it determines the target module for processing the model service request from the started Modelet model engine, wherein the Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module; based on the target module, it performs initial processing on the model service request, and further processing on the initially processed model service request based on the started local original model service provided externally through reverse proxy. The Modelet model engine is designed with a parent-child process and reverse proxy architecture. Based on the parent-child process and reverse proxy architecture, after the Modelet model engine starts, the Modelet model engine, as the parent process, starts a child process of the original model, so that the Modelet model engine can reverse proxy the original model and provide the corresponding original model service. It is understood that in this application, the Modelet engine adopts a parent-child process and reverse proxy architecture. Therefore, after the Modelet engine starts, the differences between various inference acceleration frameworks can be shielded. The Modelet engine can filter, intercept, and monitor traffic that needs to use model services. Based on the parent-child process, the child process of the model service is started. After the corresponding traffic passes through, the model service is reverse proxied, thereby achieving (framework / platform) independent, zero-intrusion model security, high availability, and tracing capabilities. It can be adapted to the deployment of any inference acceleration framework. At the same time, the reverse proxy mode can allow requests of any format to meet the needs of all model types and framework formats (after obtaining the model service request, the target module for processing the model service request is determined from the started Modelet engine based on the type of the model service request, and the model service request is initially processed, so that requests of any format can meet the requirements of the corresponding model type and format framework). This ensures the smooth operation of large model services, even in real-world enterprise applications where multiple card types and inference acceleration frameworks may coexist. The unified Modelet engine (equivalent to an intermediate layer) ensures the coexistence of multiple types and inference acceleration frameworks. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the first embodiment of the model service in this application; Figure 2 A schematic diagram of the system architecture serving the model in this application; Figure 3 A schematic diagram of the first scenario for the model service method of this application; Figure 4 A second scenario diagram illustrating the service method for the model in this application; Figure 5 A schematic diagram of the third scenario for the service method of the model in this application; Figure 6 This is a schematic diagram of the module structure of the model service device in the embodiments of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the model service in this application embodiment.

[0018] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of this application embodiment is as follows: A model service request is obtained, and the type of the model service request is determined. Based on the type of the model service request, a target module for processing the model service request is determined from the started Modelet model engine. The Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module. Based on the target module, the model service request undergoes initial processing. Furthermore, based on the original model service provided externally via a reverse proxy on the started local machine, the initially processed model service request undergoes further processing. The Modelet model engine is designed using a parent-child process and reverse proxy architecture. Based on this architecture, after the Modelet model engine starts, it starts a child process of the original model as the parent process, allowing the Modelet model engine to reverse proxy the original model and provide the corresponding original model service.

[0022] In related technologies, in actual enterprise application scenarios, there may often be situations where multiple types and multiple inference acceleration frameworks coexist, making it difficult to ensure the stable operation of large model services.

[0023] In this application, the Modelet engine adopts a parent-child process and reverse proxy architecture. Therefore, after the Modelet engine starts, the differences between various inference acceleration frameworks can be masked. The Modelet engine filters, intercepts, and monitors traffic requiring model services, and starts child processes for the model service based on the parent-child process architecture. After the corresponding traffic passes through, it reverse proxies the model service, thus achieving (framework / platform) independent, zero-intrusion model security, high availability, and tracing capabilities. It is adaptable to the deployment of any inference acceleration framework. Simultaneously, the reverse proxy mode allows requests of any format to satisfy the usage of all model types and framework formats (after obtaining a model service request, the target module for processing the request is determined from the started Modelet engine based on the type of the request, and the request is initially processed, ensuring that requests of any format meet the requirements of the corresponding model type and format framework). This ensures that in real-world enterprise applications, even with multiple GPU types and multiple inference acceleration frameworks coexisting, the unified Modelet engine (equivalent to an intermediate layer) ensures the coexistence of multiple types and multiple inference acceleration frameworks, thereby guaranteeing the stable operation of large model services.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or model service device capable of performing the above functions. The following description uses a model service device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, the embodiments of this application provide a model service, referring to Figure 1 , Figure 1 A flowchart illustrating the first embodiment serving the model of this application.

[0026] In this embodiment, the model service includes steps S10 to S30: Step S10: Obtain the model service request and determine the type of the model service request; It should be noted that the execution entity in this embodiment is the Modelet model engine, wherein, Figure 2 This is a system architecture diagram of the Modelet model engine. Figure 2 As can be seen from the diagram, upon startup, the Modelet engine starts as the parent process, correspondingly launching the original model child processes. Furthermore, the Modelet engine primarily includes a model service proxy module, a metric module, a probe module, and a built-in log service module. Specifically, the model service proxy module within the Modelet engine also includes components with various functions, such as authentication components and (distributed) tracing components, to implement authentication and tracing functions. Based on the parent-child process architecture of the Modelet engine, the model service proxy module also provides web services, meaning it can provide native model service capabilities externally through reverse proxy. For example, if model A is deployed through the Modelet engine, after the Modelet engine starts, it, as the parent process, will launch the corresponding child process, i.e., launch model A. Thus, the Modelet engine can provide native model service capabilities (i.e., the model service capabilities of model A) externally through reverse proxy.

[0027] In addition, the Modelet engine provides a Metric module for metric statistics. This Metric module can be enabled or disabled by the user. If the user chooses to enable the Metric module, the Modelet engine can allow the user to directly access the monitoring and statistics services of the model itself, such as model A (such as the Metric interface built into the MindIE framework). Furthermore, the Modelet engine can also implement customized monitoring and collection logic within the Metric module according to the user's needs, so as to better monitor the status of the deployed model service and various metrics, and return the corresponding monitoring data to the user.

[0028] Furthermore, the Modelet engine also includes a probe module, which provides an independent health detection port and two different detection methods (Direct detection and Indirect detection). In this embodiment, the Modelet engine also includes a logging module, which records log information within the Modelet engine and performs format conversion to achieve a unified log format output.

[0029] In this embodiment, after obtaining a model service request, the Modelet model engine can determine the type of the model service request. The type of the model service request can be a business request, a tracing request, a health detection request, a model status acquisition request, a model hardware usage acquisition request, etc., and is not specifically limited.

[0030] Step S20: Based on the type of the model service request, determine the target module for processing the model service request from the started Modelet model engine, wherein the Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module.

[0031] It is understood that in this embodiment, there is a mapping relationship between which module handles which type of request, and this mapping relationship is pre-existing in the Modelet engine. Therefore, the Modelet engine can selectively choose the target module to handle the model service request based on its type. The Modelet engine includes at least a model service proxy module, a metric module, a probe module, and a log service module. Of course, the Modelet engine may also include other modules, which are not specifically limited. In addition, it should be noted that the Modelet engine also has an extensible interface. Based on this extensible interface, the functionality of the Modelet engine can be increased without writing additional code, thereby improving efficiency. Furthermore, this extensible interface is associated with a preset configuration component, and the configuration component has microservice sub-components. These microservice sub-components can be combined according to requirements (and the microservice interface is hierarchical, allowing for different levels of combination) to improve the functionality and extensibility of the extensible interface.

[0032] Step S30: Based on the target module, the model service request is initially processed, and based on the original model service provided externally through a reverse proxy on the started local machine, the initially processed model service request is further processed. The Modelet model engine is designed with a parent-child process and reverse proxy architecture. Based on the parent-child process and reverse proxy architecture, after the Modelet model engine starts, the Modelet model engine starts the child process of the original model as the parent process, so that the Modelet model engine can reverse proxy the original model and provide the corresponding original model service.

[0033] In this embodiment, the initial processing of the model service request based on the target module may specifically include: Based on the target module, the model service request is forwarded. Based on the target module, the model service request is parsed and processed. Based on the target module, the model service request is processed to convert the request format; Based on the target module, the model service request is intercepted or processed.

[0034] In this embodiment, for the model service request after initial processing, if it is determined that it meets the subsequent requirements, based on the original model service provided externally by the started local machine through a reverse proxy, the model service request after initial processing is further processed. For example, if the model service request meets the authentication requirements, a large model service can be provided to the user terminal corresponding to the model service request based on the started local machine through a reverse proxy. The large model service can specifically be a query model, an expert recognition model, a classification model, or other functional models, etc., without any specific limitation.

[0035] In this embodiment, the model service request includes a business request. When the model service request is a business request, the target module includes a model service proxy module. The steps of performing initial processing on the model service request based on the target module, and performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: Step S31: Determine the link context corresponding to the service request; In this embodiment, the model service request includes a business request, which can be a loan business request, a credit business request, etc., and is not specifically limited. A link context can be generated based on the business request (implemented through a distributed link component). The method for generating the link context based on the business request can be: Method 1: Generate a link context based on the address information in the packet header of the business request; Method 2: Generate a link context based on the probe path information carried in the business request.

[0036] Step S32: Based on the authentication component in the model service proxy module, load and decrypt the authentication information configuration from the preset environment variable configuration to verify the link context; After determining the link context corresponding to the service request, such as Figure 3 As shown, based on the authentication component in the model service proxy module, the authentication information configuration is loaded and decrypted from the preset environment variable configuration to verify the link context. The preset environment variable configuration is pre-configured, and the preset environment variable configuration is different for different model services. After loading and decrypting the authentication information configuration of the corresponding model from the preset environment variable configuration, it is compared with the link context to determine whether the two are consistent.

[0037] Step S33: If the verification passes, based on the original model service provided externally by the local machine through a reverse proxy, perform other processing on the initially processed model service request.

[0038] If the model's authentication information configuration is inconsistent with the link context, the verification will fail. If the model's authentication information configuration is consistent with the link context, the verification will pass. If the verification passes, based on the original model service provided externally through a reverse proxy on the local machine, the initially processed model service request will undergo further processing.

[0039] It should be noted that the model services deployed by current traditional inference acceleration frameworks lack authentication and authorization mechanisms. If the source address of model resources is leaked, on the one hand, users can bypass the service provider's control and directly access model resources. Such illegal traffic may consume a large amount of computing resources without permission, thus affecting the stability of the model service and causing the model to crash. On the other hand, directly exposed services may be attacked by malicious traffic, even leading to the paralysis of infrastructure.

[0040] In this embodiment, the authentication component interceptor is enabled by default within the model service proxy module, thereby enabling the authentication function of access traffic. In addition, the distributed link tracing component interceptor is also enabled within the model service proxy module (where the Modelet engine complies with the OpenTracing protocol specification), thereby enabling the link log reporting function. Therefore, this application can enable the model service to be effectively observed in a huge and complex distributed call link, avoiding greater operation and maintenance costs.

[0041] In this embodiment, the original model service provided locally via a reverse proxy after startup can specifically be: If the original model service is service 'a', then the Modelet model engine provides service 'a' to the outside world through a reverse proxy. If the original model service is service b, then the Modelet model engine provides service b to the outside world through a reverse proxy.

[0042] Service A and Service B can be specifically determined based on the type of the large model. The target module includes a log service module. If the verification passes, after further processing of the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, the process includes: Step A1: If the verification passes, generate the first link information corresponding to the verified link. Step A2: If the verification fails, generate the second link information corresponding to the verified link. Step A3: The first link information and the second link information are sent to the log service module in a preset asynchronous manner for the log service module to save and output; wherein, the log service module also intercepts and processes the traffic output from the original model through the parent and child processes of the Modelet model engine and the reverse proxy architecture.

[0043] In this embodiment, as Figure 2 As shown, if the verification passes, the first link information corresponding to the verified link is generated; if the verification fails, the second link information corresponding to the verified link is generated (the two paths are completely different). The first and second link information are then sent to the log service module in a preset asynchronous manner (different processes) for the log service module to save and output. Since different link information is saved, the model service can be effectively observed in the huge and complex distributed call chain. It should be noted that in this embodiment, the log service module also intercepts traffic output from the original model through the parent and child processes of the Modelet model engine and the reverse proxy architecture. That is, the model service deployed by traditional inference acceleration frameworks lacks authentication and authorization mechanisms. If the source address of the model resource is leaked, on the one hand, users can bypass the service provider's control and directly access the model resource. Such illegal traffic may consume a large amount of computing resources without permission, thus affecting the stability of the model service and causing the model to crash; on the other hand, the directly exposed service may be attacked by malicious traffic, even leading to the paralysis of the infrastructure. In this embodiment, the Modelet engine directly intercepts the traffic output from the original model, preventing it from being attacked by malicious traffic or even causing the infrastructure to be paralyzed.

[0044] In other words, it's important to note that in the cloud era, monolithic applications struggle to support the development of large or even ultra-large applications, leading to the emergence of distributed application architectures. However, this also brings significant operational challenges, making troubleshooting and maintenance heavily reliant on distributed call tracing capabilities. Current inference acceleration frameworks lack this capability, making it impossible to effectively observe model services within massive and complex distributed call chains, resulting in higher operational costs. In this embodiment, by asynchronously storing the first and second call chain information, the model service can be effectively observed within these massive and complex distributed call chains.

[0045] Compared to related technologies, in actual enterprise applications, there are often situations where multiple types and multiple inference acceleration frameworks coexist, making it difficult to ensure the stable operation of large model services. This application obtains model service requests and determines the type of the model service request; based on the type of the model service request, it determines the target module for processing the model service request from the started Modelet model engine, wherein the Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module; based on the target module, it performs initial processing on the model service request, and further processing on the initially processed model service request based on the started local original model service provided externally through reverse proxy. The Modelet model engine is designed with a parent-child process and reverse proxy architecture. Based on the parent-child process and reverse proxy architecture, after the Modelet model engine starts, the Modelet model engine, as the parent process, starts a child process of the original model, so that the Modelet model engine can reverse proxy the original model and provide the corresponding original model service. It is understood that in this application, the Modelet engine adopts a parent-child process and reverse proxy architecture. Therefore, after the Modelet engine starts, the differences between various inference acceleration frameworks can be shielded. The Modelet engine can filter, intercept, and monitor traffic that needs to use model services. Based on the parent-child process, the child process of the model service is started. After the corresponding traffic passes through, the model service is reverse proxied, thereby achieving (framework / platform) independent, zero-intrusion model security, high availability, and tracing capabilities. It can be adapted to the deployment of any inference acceleration framework. At the same time, the reverse proxy mode can allow requests of any format to meet the needs of all model types and framework formats (after obtaining the model service request, the target module for processing the model service request is determined from the started Modelet engine based on the type of the model service request, and the model service request is initially processed, so that requests of any format can meet the requirements of the corresponding model type and format framework). This ensures the smooth operation of large model services, even in real-world enterprise applications where multiple card types and inference acceleration frameworks may coexist. The unified Modelet engine (equivalent to an intermediate layer) ensures the coexistence of multiple types and inference acceleration frameworks.

[0046] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4The model service request includes a metric collection request. When the model service request is a metric collection request, the target module includes a Metric module. Prior to the step of performing initial processing on the model service request based on the target module, the following steps are included: Determine whether the Metric module for metric statistics is enabled on the user's device; In this embodiment, the user client can enable or disable the Metric module used for metric statistics; that is, this is determined by the user client.

[0047] If enabled, the steps of initially processing the model service request based on the Metric module, and further processing the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: The configuration information determined by the user terminal is determined, and based on the configuration information, the target statistical functions of the Metric module that need to be started are determined; wherein, the target statistical functions include Metric interface functions, monitoring and statistical service functions that directly access the model itself, and preset customized monitoring and collection related logic functions; In this embodiment, if the Metric module is turned off, the subsequent steps are not executed. If the Metric module is turned on, the configuration information determined by the user terminal is determined. Based on the configuration information, the target statistical function of the Metric module that needs to be started is determined. For example, the configuration information may be the status of the model that needs to be statistically analyzed, the hardware information of the model, or the memory and graphics card usage, etc.

[0048] Overall, the target statistical function of the Metric module can be specifically: allowing users to directly access the model's own monitoring and statistical services (such as the Metric interface built into the MindIE framework), or implementing more customized monitoring and collection logic internally to better monitor the model service status and various indicators.

[0049] Based on the initiated target statistics function, the indicator collection request is initially processed, and based on the initiated local original model service provided externally through a reverse proxy, the initially processed indicator collection request is further processed.

[0050] In this embodiment, the indicator collection request is initially processed based on the initiated target statistical function, and further processing is performed on the initially processed indicator collection request based on the initiated original model service provided externally through a reverse proxy.

[0051] In this embodiment, the steps of determining whether the Metric module for metric statistics is enabled on the user terminal, and if enabled, performing initial processing on the model service request based on the Metric module, and further processing on the initially processed model service request based on the locally provided original model service via reverse proxy, include: determining the configuration information determined by the user terminal; determining the target statistical function of the Metric module to be enabled based on the configuration information; wherein the target statistical function includes Metric interface function, monitoring and statistical service function that directly accesses the model itself, and preset customized monitoring and collection related logic function; performing initial processing on the metric collection request based on the enabled target statistical function, and further processing on the initially processed metric collection request based on the locally provided original model service via reverse proxy. In this embodiment, the model's status, hardware information, and memory / GPU usage can be collected to meet the user terminal's metric collection needs.

[0052] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 The model service request includes a health detection request. When the model service request is a health detection request, the target module includes a probe module. The steps of performing initial processing on the model service request based on the target module, and performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: Determine the configuration information on the user's end, and determine the health detection method based on the configuration information; Based on the health detection method, a probe mode is determined according to the health detection interface determined by the target module. Based on the detection mode, the model service request is initially processed, and based on the original model service provided externally by the local system through a reverse proxy, the initially processed model service request is further processed.

[0053] In this embodiment, the configuration information of the user terminal is determined, and the health detection method is determined based on the configuration information. Depending on whether the health detection method is a Direct health detection method or an inDirect health detection method, a probe mode is determined based on the health detection interface determined by the target module. Based on the detection mode, the model service request is initially processed (the initial processing method is different for different probe modes), and based on the original model service provided externally by the started local reverse proxy, the initially processed model service request is further processed.

[0054] The steps of initially processing the model service request based on the detection mode, and then further processing the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: If the detection mode is Direct health detection mode, the model service request is forwarded directly, and the original model service provided to the outside world through the reverse proxy on the local machine is further processed. If the detection mode is inDirect health detection, the probe module performs advanced processing on the model service request. Based on the original model service provided externally via reverse proxy, the advanced processing of the model service request is further processed. The advanced processing includes converting the model service request into a request that meets the requirements of the original model service.

[0055] In current technologies, when providing model services externally, robust protection mechanisms are needed for the models themselves. That is, unlike traditional web services, user input significantly impacts model resource consumption. Ordinary load balancing cannot guarantee stability at the model instance level, and current open-source frameworks lack standardized implementations of health checks, failing to accurately reflect the specific load conditions of the models. Furthermore, most open-source frameworks do not provide graceful exit mechanisms. When handling model upgrades, resource scaling, instance migration, etc., manual intervention through higher-level gateways and load balancers is required to achieve seamless traffic switching, resulting in higher manpower and resource costs and greater operational risks. In this embodiment, it should be noted that a health monitoring method is provided. Specifically, if the detection mode is the Direct health detection mode, the model service request is directly forwarded. That is, the user directly inputs a request that can be directly forwarded into the Modelet model engine. This requires the user to construct complex inputs so that the corresponding request meets the requirements.

[0056] In other words, Direct health checks are a flexible detection mode. In this mode, the Modelet engine forwards received health probe requests unchanged to the corresponding underlying interface according to the user-configured path. The user client can use this to implement custom judgment logic or reuse interfaces provided by the original inference acceleration framework. Based on this, the specific load situation under the Direct health check mode can be reflected.

[0057] If the detection mode is inDirect health detection, the probe module performs advanced processing on the model service request, such as format conversion or directly obtaining the model's input requirements. Based on the model's input requirements, the model service request is then processed in an advanced manner.

[0058] That is, in this embodiment, other interfaces or built-in health detection scripts (such as MindIE) besides the Direct health detection method are also provided. Therefore, in the inDirect health detection method, the access health detection interface of the Modelet model engine in this embodiment will generate a simple, single-token inference request internally (the default path is / v1 / chat / completions).

[0059] It should be noted that, in practice, some interfaces or built-in scripts have been found to remain accessible even when the model is unavailable, and health detection scripts may also function normally even when the model appears to be unresponsive. Based on these considerations, this embodiment assumes that the model inference interface, as a core service provided to users, accurately reflects the actual inference load of the current model instance through the success or failure of requests and the duration of latency.

[0060] Based on the above analysis, this embodiment, through actual testing, shows that during health probes, although peak throughput may be reduced by about 1% in extreme cases (triggering a health probe timeout, causing Kubernetes to remove traffic from the instance), the model itself is almost unable to provide normal inference services at this point. Therefore, it can better protect the model and prevent excessive inference load from causing model downtime. In normal use, a configuration of probing every 30 seconds with a 10-second timeout is recommended. Based on testing and practical experience, this configuration accurately and promptly reflects the load situation without affecting daily business access.

[0061] In this embodiment, the probe module also provides a graceful exit interface, which provides a simple sleep waiting method by default, and can also be extended to achieve more accurate exit detection by detecting the status of the HTTP connection thread.

[0062] In this embodiment, the configuration information of the user terminal is determined, and the health detection method is determined based on the configuration information. Based on the health detection method and the health detection interface determined by the target module, a probe mode is determined. Based on the detection mode, the model service request is initially processed, and based on the original model service provided externally via a reverse proxy, further processing is performed on the initially processed model service request. In this embodiment, the function of health detection can be implemented.

[0063] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the model service of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0064] This application also provides a model service device, please refer to... Figure 6 Applied to the Modelet model engine, the model service device includes: The acquisition module 10 is used to acquire model service requests and determine the type of the model service requests; The determination module 20 is used to determine the target module for processing the model service request from the started Modelet model engine based on the type of the model service request, wherein the Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module. The proxy module 30 is used to perform initial processing on the model service request based on the target module, and to perform other processing on the initially processed model service request based on the original model service provided externally through a reverse proxy on the started local machine. The Modelet model engine is designed with a parent-child process and reverse proxy architecture. Based on the parent-child process and reverse proxy architecture, after the Modelet model engine starts, the Modelet model engine starts the child process of the original model as the parent process, so that the Modelet model engine can reverse proxy the original model and provide the corresponding original model service.

[0065] In one embodiment, the model service device is used to implement: Determine the link context corresponding to the service request; Based on the authentication component in the model service proxy module, the authentication information configuration is loaded from the preset environment variable configuration and decrypted to verify the link context; If the verification passes, based on the original model service provided externally via reverse proxy on the local machine, the initially processed model service request is further processed.

[0066] In one embodiment, the model service device is used to implement: If the verification passes, the first link information corresponding to the verified link is generated; If the verification fails, the second link information corresponding to the verified link is generated; The first link information and the second link information are sent to the log service module in a preset asynchronous manner for the log service module to save and output; wherein, the log service module also intercepts and processes the traffic output from the original model through the parent and child processes of the Modelet model engine and the reverse proxy architecture.

[0067] In one embodiment, the model service device is used to implement: Determine whether the Metric module for metric statistics is enabled on the user's device; If enabled, the steps of initially processing the model service request based on the Metric module, and further processing the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: The configuration information determined by the user terminal is determined, and based on the configuration information, the target statistical functions of the Metric module that need to be started are determined; wherein, the target statistical functions include Metric interface functions, monitoring and statistical service functions that directly access the model itself, and preset customized monitoring and collection related logic functions; Based on the initiated target statistics function, the indicator collection request is initially processed, and based on the initiated local original model service provided externally through a reverse proxy, the initially processed indicator collection request is further processed.

[0068] In one embodiment, the model service device is used to implement: Determine the configuration information on the user's end, and determine the health detection method based on the configuration information; Based on the health detection method, a probe mode is determined according to the health detection interface determined by the target module. Based on the detection mode, the model service request is initially processed, and based on the original model service provided externally by the local system through a reverse proxy, the initially processed model service request is further processed.

[0069] In one embodiment, the model service device is used to implement: If the detection mode is Direct health detection mode, the model service request is forwarded directly, and the original model service provided to the outside world through the reverse proxy on the local machine is further processed. If the detection mode is inDirect health detection, the probe module performs advanced processing on the model service request. Based on the original model service provided externally via reverse proxy, the advanced processing of the model service request is further processed. The advanced processing includes converting the model service request into a request that meets the requirements of the original model service.

[0070] The model service apparatus provided in this application, employing the model service described in the above embodiments, can solve the technical problems of model services. Compared with the prior art, the beneficial effects of the model service apparatus provided in this application are the same as those of the model service provided in the above embodiments, and other technical features in the model service apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0071] This application provides a model service device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the model service in the above embodiment 1.

[0072] The following is for reference. Figure 7 The diagram illustrates a structural schematic of a model service device suitable for implementing embodiments of this application. The model service device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, tablets, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The model service device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0073] like Figure 7As shown, the model service device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the model service device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the model service device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a model service device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented alternatively.

[0074] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0075] The model service device provided in this application, employing the model service described in the above embodiments, can solve the technical problems of model services. Compared with the prior art, the beneficial effects of the model service device provided in this application are the same as those of the model service provided in the above embodiments, and other technical features of the model service device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0076] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0077] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0078] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the model service in the above embodiments.

[0079] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0080] The aforementioned computer-readable storage medium may be included in the model service device; or it may exist independently and not assembled into the model service device.

[0081] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the model service device, the model service device: collects access information from different application sources through an access collection program and stores the access information in a message queue, wherein the access collection program is deployed in the runtime environment of the application; performs service call dependency deduction on the access information in the message queue to obtain the call relationship between the different applications; and generates an application topology architecture based on the call relationship.

[0082] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0085] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for performing the above-described model service, and is capable of solving the technical problems of the model service. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the model service provided in the above embodiments, and will not be repeated here.

[0086] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the model service as described above.

[0087] The computer program product provided in this application can solve the technical problems of model services. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the model services provided in the above embodiments, and will not be repeated here.

[0088] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A model service method, characterized in that, Applied to the Modelet model engine, the model service method includes: Obtain the model service request and determine the type of the model service request; Based on the type of the model service request, the target module for processing the model service request is determined from the started Modelet model engine, wherein the Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module; Based on the target module, the model service request is initially processed, and based on the original model service provided externally through a reverse proxy on the started local machine, the initially processed model service request is further processed. The Modelet model engine is designed with a parent-child process and reverse proxy architecture. Based on the parent-child process and reverse proxy architecture, after the Modelet model engine starts, the Modelet model engine, as the parent process, starts the child process of the original model so that the Modelet model engine can reverse proxy the original model and provide the corresponding original model service.

2. The model service method as described in claim 1, characterized in that, The model service request includes a business request. When the model service request is a business request, the target module includes a model service proxy module. The steps of performing initial processing on the model service request based on the target module, and performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: Determine the link context corresponding to the service request; Based on the authentication component in the model service proxy module, the authentication information configuration is loaded from the preset environment variable configuration and decrypted to verify the link context; If the verification passes, based on the original model service provided externally via reverse proxy on the local machine, the initially processed model service request is further processed.

3. The model service method as described in claim 2, characterized in that, The target module includes a log service module. Following the step of performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy when the verification passes, the process includes: If the verification passes, the first link information corresponding to the verified link is generated; If the verification fails, the second link information corresponding to the verified link is generated; The first link information and the second link information are sent to the log service module in a preset asynchronous manner for the log service module to save and output; wherein, the log service module also intercepts and processes the traffic output from the original model through the parent and child processes of the Modelet model engine and the reverse proxy architecture.

4. The model service method as described in claim 1, characterized in that, The model service request includes a metric collection request. When the model service request is a metric collection request, the target module includes a Metric module. Before the step of performing initial processing on the model service request based on the target module, the following steps are included: Determine whether the Metric module for metric statistics is enabled on the user's device. If enabled, the steps of initially processing the model service request based on the Metric module, and further processing the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: The configuration information determined by the user terminal is determined, and based on the configuration information, the target statistical functions of the Metric module that need to be started are determined; wherein, the target statistical functions include Metric interface functions, monitoring and statistical service functions that directly access the model itself, and preset customized monitoring and collection related logic functions; Based on the initiated target statistics function, the indicator collection request is initially processed, and based on the initiated local original model service provided externally through a reverse proxy, the initially processed indicator collection request is further processed.

5. The model service method as described in claim 1, characterized in that, The model service request includes a health probe request. When the model service request is a health probe request, the target module includes a probe module. The steps of performing initial processing on the model service request based on the target module, and performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: Determine the configuration information on the user's end, and determine the health detection method based on the configuration information; Based on the health detection method, a probe mode is determined according to the health detection interface determined by the target module. Based on the detection mode, the model service request is initially processed, and based on the original model service provided externally by the local system through a reverse proxy, the initially processed model service request is further processed.

6. The model service method as described in claim 5, characterized in that, The steps of initially processing the model service request based on the detection mode, and then performing further processing on the initially processed model service request based on the original model service provided externally via a reverse proxy on the started local machine, include: If the detection mode is Direct health detection mode, the model service request is forwarded directly, and the original model service provided to the outside world through the reverse proxy on the local machine is further processed. If the detection mode is inDirect health detection, the probe module performs advanced processing on the model service request. Based on the original model service provided externally via reverse proxy, the advanced processing of the model service request is further processed. The advanced processing includes converting the model service request into a request that meets the requirements of the original model service.

7. A model service device, characterized in that, Applied to the Modelet model engine, the model service device includes: The acquisition module is used to acquire model service requests and determine the type of the model service requests; The determination module is used to determine the target module for processing the model service request from the started Modelet model engine based on the type of the model service request, wherein the Modelet model engine includes at least a model service proxy module, a metric module, a probe module, and a log service module. The proxy module is used to perform initial processing on the model service request based on the target module, and to perform other processing on the initially processed model service request based on the original model service provided externally through a reverse proxy on the started local machine. The Modelet model engine is designed with a parent-child process and reverse proxy architecture. Based on the parent-child process and reverse proxy architecture, after the Modelet model engine starts, the Modelet model engine starts the child process of the original model as the parent process, so that the Modelet model engine can reverse proxy the original model and provide the corresponding original model service.

8. A model service device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model service as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the model service as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the model service as described in any one of claims 1 to 6.

Images