Model instance monitoring method and device for container platform, electronic equipment and storage medium

Abstract

The application discloses a kind of container platform model instance monitoring method, device, electronic equipment and storage medium, it is related to the model instance management field of container, comprising: when target model instance in container is in preset state, according to the basic running environment state, heartbeat level information, inference behavior characteristics and inference result characteristics of target model instance, generate behavior characteristic vector;According to behavior characteristic vector and comprehensive determination model, determine the confidence of the multiple operating states corresponding to target model instance;According to the confidence corresponding to multiple operating states, determine the actual operating state of target model instance;If actual operating state is abnormal state, then target model instance is recovered or recycled.It is realized according to behavior characteristic vector accurately to determine the actual operating state of target model instance, complete the monitoring of model instance working condition in container.Can be in time to abnormal target model instance is recovered and recycled, improve the reliability of model instance, improve resource utilization.

Description

Technical Field

[0001] This invention relates to the field of container-related model instance management, and more particularly to model instance monitoring methods, apparatus, electronic devices, and storage media for container platforms. Background Technology

[0002] With the widespread application of artificial intelligence technology in the industrial field, more and more industrial models are being deployed on servers or edge nodes in a service-oriented manner. The model's operating environment is encapsulated through containerization technology, and interface services are provided to upper-layer business systems.

[0003] Current management tools are container-oriented, such as the allocation and reclamation of container resources. However, they cannot monitor model instances set up inside containers, leading to a situation where abnormal model instances cannot be handled promptly when the container is running normally. Improving the monitoring efficiency of model instances is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for monitoring model instances in a container platform, in order to solve the problem that model instances in containers cannot be monitored.

[0005] According to one aspect of the present invention, a method for monitoring model instances on a container platform is provided, comprising:

[0006] When the target model instance in the container is in a preset state, a behavior feature vector is generated based on the basic runtime environment state, heartbeat level information, reasoning behavior characteristics, and reasoning result characteristics of the target model instance.

[0007] The confidence levels corresponding to multiple running states of the target model instance are determined based on the behavioral feature vector and the comprehensive judgment model.

[0008] The actual running state of the target model instance is determined based on the confidence levels corresponding to the multiple running states.

[0009] If the actual operating state is abnormal, the target model instance will be restored or recycled.

[0010] According to another aspect of the present invention, a model instance monitoring device for a container platform is provided, comprising:

[0011] The behavior feature vector generation module is used to generate a behavior feature vector based on the basic runtime environment state, heartbeat level information, inference behavior features and inference result features of the target model instance when the target model instance in the container is in a preset state.

[0012] The running state prediction module is used to determine the confidence level corresponding to multiple running states of the target model instance based on the behavioral feature vector and the comprehensive judgment model;

[0013] The actual running status determination module is used to determine the actual running status of the target model instance based on the confidence levels corresponding to the multiple running states.

[0014] An exception handling module is used to restore or reclaim the target model instance if the actual running state is an abnormal state.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the model instance monitoring method of the container platform according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the model instance monitoring method of the container platform according to any embodiment of the present invention.

[0020] The technical solution of this invention, when a target model instance in a container is in a preset state, generates a behavioral feature vector based on the target model instance's basic operating environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics; determines the confidence level corresponding to multiple operating states of the target model instance based on the behavioral feature vector and a comprehensive judgment model; determines the actual operating state of the target model instance based on the confidence level corresponding to the multiple operating states; and if the actual operating state is an abnormal state, restores or reclaims the target model instance. Compared to the current inability to monitor model instances in containers, the technical solution provided by this invention can generate a behavioral feature vector based on the target model instance's basic operating environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics in a preset state. The behavioral feature vector can comprehensively represent the operating condition of the target model instance in vector form. The confidence level corresponding to multiple operating states of the target model instance is determined based on the behavioral feature vector and a comprehensive judgment model; the actual operating state of the target model instance is determined based on the confidence level corresponding to the multiple operating states. This achieves accurate determination of the actual operating state of the target model instance based on the behavioral feature vector, completing the monitoring of the operating condition of model instances in containers. Based on this, if the actual operating state is abnormal, the target model instance will be restored or recycled in a timely manner, thereby improving the reliability of the model instance and increasing resource utilization.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a model instance monitoring method for a container platform provided in an embodiment of the present invention;

[0024] Figure 2 A flowchart illustrating another method for monitoring model instances on a container platform provided in an embodiment of the present invention;

[0025] Figure 3 A flowchart illustrating another method for monitoring model instances on a container platform provided in an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of the structure of a model instance monitoring device for a container platform provided in an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of the structure of an electronic device for implementing the model instance monitoring method of the container platform in this embodiment of the invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0030] The inventors discovered that with the widespread application of artificial intelligence technology in the industrial field, more and more industrial models (such as prediction models, detection models, optimization models, etc.) are deployed on servers or edge nodes in a service-oriented manner. The model's operating environment is encapsulated through containerization technology (such as Docker) and interface services are provided to upper-layer business systems.

[0031] Model services are typically managed in the following ways: (1) Starting and stopping model containers directly on container platforms (such as Docker and Kubernetes). (2) Determining model availability by checking container running status (running / stopped). (3) Maintaining the model service manually or through scheduled tasks.

[0032] However, since a container being in a running state does not equate to the model service being available, situations such as model process freezes, interface errors, and unreachable ports cannot be accurately reflected by the container status, making the judgment of model running status inaccurate. The above approach focuses more on the container lifecycle than the business lifecycle of the model instance, lacking comprehensive lifecycle management for the model and reasonable resource management.

[0033] It is evident that using container runtime status or resource usage as the sole criterion for determining model runtime status is insufficient to identify situations where the container is running normally but the model's inference logic is malfunctioning or its output is distorted. This fails to meet the reliability requirements of industrial scenarios. Management tools are container-oriented, such as the allocation and reclamation of container resources. However, they cannot monitor the model instances set up within the container, leading to a situation where abnormal model instances cannot be addressed promptly even when the container is running normally. Therefore, improving the monitoring efficiency of model instances is a pressing issue that needs to be addressed.

[0034] Figure 1 This is a flowchart illustrating a model instance monitoring method for a container platform provided in an embodiment of the present invention. This embodiment is applicable to situations involving the monitoring and management of model instances in containers. The method can be executed by a model instance monitoring device for the container platform, which can be implemented in hardware and / or software. This model instance monitoring device can be configured in electronic devices such as personal computers and servers. Figure 1 As shown, the method includes:

[0035] Step S101: When the target model instance in the container is in a preset state, generate a behavior feature vector based on the basic operating environment state, heartbeat level information, reasoning behavior characteristics and reasoning result characteristics of the target model instance.

[0036] Optionally, before generating the behavior feature vector based on the target model instance's basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics when the target model instance in the container is in a preset state, the method further includes:

[0037] Configure the state machine of the target model instance, the state machine including initialization state, startup state, model warm-up state, running state, degenerate running state, abnormal state, stopped state and recycled state.

[0038] This invention provides a state machine for model instances, enabling fine-grained management of the state of a model instance throughout its entire runtime. Unlike management methods based solely on container runtime states, this invention's lifecycle state machine focuses on the business operation behavior of the model instance, accurately reflecting its true availability. Figure 2 This is a schematic diagram of a model instance state transition provided in an embodiment of the present invention. The state machine of the model instance includes at least the following states:

[0039] (1) Initialization state (S0): The model instance has been created but has not yet been allocated running resources. It is in the parameter configuration and environment preparation stage.

[0040] (2) Startup status (S1): The model instance has triggered the startup process and is completing operations such as container creation and runtime environment loading, but has not yet provided model services to the outside world.

[0041] (3) Model warm-up state (S1-1): The container corresponding to the model instance has been successfully started and the model service port is in the listening state, but the model has not yet completed model-level initialization operations such as weight loading, computation graph construction, cache initialization or inference engine warm-up. At this time, the model instance cannot provide stable inference services to the outside world.

[0042] (4) Running status (S2): The model instance has completed the preheating and passed the model-level availability test, and can provide model inference or computation services to the outside world in accordance with the predetermined service quality.

[0043] (5) Degraded running state (S2-1): The model instance is still running, but abnormal model inference delay, limited resources, or degraded input data quality are detected. The model instance is switched to low precision, rate limiting, or simplified calculation mode to ensure business continuity.

[0044] (6) Abnormal state (S3): The model instance exhibits abnormal running behavior, including but not limited to abnormal model inference results, failure of internal model logic, and model response timeout. Although the corresponding container is still running, the model instance no longer meets the business usage requirements.

[0045] (7) Stopped state (S4): The model instance has received a stop or destroy command and is performing container stop, model unloading and resource release operations.

[0046] (8) Reclaimed status (S5): The model instance has completed resource reclamation, and the port resources, computing resources and storage resources it occupies have been released.

[0047] The state machine of a model instance can avoid the problem of inconsistency between the model instance state and the actual business availability. The state of a model instance can be accurately defined through the above-mentioned states. The target model instance is any model instance in the container platform. The container platform runs multiple containers, each running one or more model instances. A model instance can also be split into multiple processing layers, each deployed in a different container.

[0048] Accordingly, when the target model instance in the container is in a preset state, a behavioral feature vector is generated based on the target model instance's basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics, including:

[0049] When the target model instance in the container is in a running state or a degraded running state, a behavior feature vector is generated based on the basic running environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance.

[0050] The above implementation method configures a state machine for model instances. By defining multiple operating states of model instances, it enables standardized management of the working conditions of model instances, orderly definition of the life cycle of model instances, and improves the management efficiency of model instances.

[0051] Optionally, a behavioral feature vector is generated based on the basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance, including:

[0052] The basic runtime environment status of the target model instance is obtained. This basic runtime environment status includes: the container's running status, the container's CPU usage information, the container's memory usage information, the container's GPU usage information, and its network connection status. This basic runtime environment status can be used for preliminary detection to rule out basic anomalies at the container level.

[0053] Optionally, network reachability checks are performed on the model service port to determine whether the model instance has the ability to communicate externally through active probing or handshake requests. In one embodiment, the model service port can be checked by sending a health check request or an inference test request. When the port is detected as unreachable or the response is abnormal, the corresponding abnormal flag is recorded.

[0054] The heartbeat level information of the target model instance is obtained. This heartbeat level information includes the following: the version identifier of the target model instance, the loading status of the target model instance, the current number of service requests for the target model instance, the most recent inference response time of the target model instance, or the exception tag of the target model instance. The heartbeat level information reflects the internal operating status of the target model instance, enabling real-time awareness of its internal operation.

[0055] The inference behavior characteristics of the target model instance are obtained, including: inference response time and its fluctuation characteristics, inference request queue length, inference success rate, and inference request throughput per unit time. These inference behavior characteristic data represent the operational behavior characteristics of the target model instance during the inference execution phase, reflecting the performance of the target model instance in the actual service process.

[0056] The inference result features of the target model instance are obtained, including: the distribution characteristics of the inference output results, the trend of inference result changes, the stability index of the inference results, or the degree of deviation of the inference results from historical operating data. These inference result features are used to analyze the stability and effectiveness of the model output results.

[0057] The basic operating environment state, heartbeat level information, reasoning behavior characteristics, and reasoning result characteristics of the target model instance are digitally mapped to obtain corresponding single-dimensional vectors; multiple single-dimensional vectors are combined to obtain a behavior feature vector.

[0058] For example, model instance behavior feature vectors include, but are not limited to, the following feature types:

[0059] (1) Operating environment characteristics, including CPU utilization, memory usage, GPU utilization, network connectivity status, etc. (2) Model service reachability characteristics, including model service port connectivity, health check response status, etc. (3) Model running status characteristics, including model loading status, model-level heartbeat interval, current service request count, etc. (4) Inference behavior performance characteristics, including inference response time, inference success rate, request queue length, inference throughput per unit time, etc. (5) Inference result characteristics, including inference result distribution characteristics, inference result change trend, inference output stability index, and deviation from historical operating data, etc.

[0060] The aforementioned features collectively constitute a multi-dimensional feature space for describing the operational behavior of model instances, thus forming a model instance behavior feature vector. The construction of this behavior feature vector primarily relies on model service quality indicators, model operational stability indicators, and model inference result stability indicators, thereby enabling the characterization of the model instance's operational state from three dimensions: the resource layer, the service layer, and the inference logic layer.

[0061] The above implementation can generate behavioral feature vectors through basic operating environment status, heartbeat level information, inference behavior characteristics and inference result characteristics, so as to accurately calculate behavioral feature vectors and enable behavioral feature vectors to more accurately represent the working conditions of target model instances.

[0062] Step S102: Determine the confidence level corresponding to multiple running states of the target model instance based on the behavioral feature vector and the comprehensive judgment model.

[0063] Furthermore, before determining the confidence levels corresponding to multiple running states of the target model instance based on the behavioral feature vector and the comprehensive judgment model, the method further includes:

[0064] Anomalies are determined based on preset rules for heartbeat-level information and inference behavior characteristics. These preset rules include: determining whether anomalies occur based on inference response time and timeout threshold; determining whether anomalies occur based on inference success rate and success rate threshold; and determining whether anomalies occur based on whether heartbeat-level information is interrupted.

[0065] The preset rules include, but are not limited to: 1. Determining whether something is abnormal based on the inference response time and timeout threshold. If the inference response time is greater than the timeout threshold, it is considered abnormal; otherwise, it is considered normal.

[0066] 2. Determine if an anomaly is detected based on the inference success rate and a success rate threshold. If the inference success rate is less than the success rate threshold, it is considered an anomaly; otherwise, it is considered normal.

[0067] 3. Determine if there is an abnormality based on whether the heartbeat level information is interrupted. If the heartbeat level information is interrupted, it is abnormal; otherwise, it is considered normal.

[0068] The above implementation method can judge the basic operating status of model instances based on relevant thresholds through preset rules, realize the rapid identification of model instance anomalies, and improve the efficiency of anomaly identification.

[0069] Optionally, the confidence levels corresponding to multiple running states of the target model instance are determined based on the behavioral feature vector and the comprehensive judgment model, including:

[0070] A training set is pre-acquired, comprising training behavior feature vectors and running status labels. Using this training set, an initial model is trained using a pre-defined training method to obtain a comprehensive judgment model. During training, a training dataset is constructed based on historical running data. Each training sample includes a behavior feature vector and a corresponding running status label. The comprehensive judgment model is trained using a random forest algorithm.

[0071] In use, the feature vector is input into the comprehensive judgment model, which is used to perform state machine classification based on the feature input vector and output the confidence level of the target model instance in the abnormal state and the degenerate running state, respectively.

[0072] When in use, the comprehensive decision model predicts the real-time feature vector. The comprehensive decision model outputs the running state of the target model instance and its confidence level (probability value).

[0073] The above implementation method can automatically identify the running state of a target model instance based on behavioral feature vectors, using a trained comprehensive judgment model. By using the confidence levels of different running states, the running state of the target model instance can be evaluated more comprehensively, improving the reliability of running state identification.

[0074] Step S103: Determine the actual running state of the target model instance based on the confidence levels corresponding to the multiple running states.

[0075] Optionally, determining the actual running state of the target model instance based on the confidence levels corresponding to the multiple running states includes:

[0076] The confidence levels of abnormal states and degraded operating states are weighted and summed according to preset weighting parameters to obtain the actual confidence level; the actual operating state of the target model instance is determined based on the actual confidence level and the preset confidence threshold.

[0077] Actual confidence level = P(S3) + α × P(S2-1)

[0078] Here, α is the weighting parameter. P(S3) represents the probability that the target model instance is judged to be in an abnormal state, and P(S2-1) represents the probability that the target model instance is in a degraded operating state. This probability is also called the confidence level, which is output by the comprehensive judgment model. By weighting and combining the probability of abnormal state with the probability of degraded operating state, the sensitivity to potential abnormal trends can be improved, thereby identifying performance degradation or abnormal operation of model instances in advance.

[0079] In one implementation, the final running state of a model instance can be determined by comparing the anomaly confidence score with a preset threshold, and the corresponding state machine transition or alarm processing logic can be triggered.

[0080] For example, determining the running status of a model instance based on anomaly confidence scores can be implemented as follows:

[0081] When the anomaly confidence level is greater than the first threshold, it is determined to be an abnormal state (S3).

[0082] When the anomaly confidence level is between the first threshold and the second threshold, it is determined to be a degraded operating state (S2-1).

[0083] When the anomaly confidence level is lower than the second threshold, it is determined to be in the running state (S2).

[0084] The above implementation method can statistically set weights to flexibly identify the actual confidence level and improve the accuracy of identification.

[0085] Step S104: If the actual operating state is abnormal, then the target model instance is restored or recycled.

[0086] Optionally, if the actual operating state is abnormal, the target model instance is restored or recycled, including:

[0087] If the target model instance is in an abnormal state, it will be restored according to a preset recovery strategy. The preset recovery strategy includes: model parameter level reload strategy, model running mode switching strategy, model instance reconstruction strategy, or model instance restart strategy.

[0088] When the comprehensive judgment model determines that the target model instance is in an abnormal state, any of the following strategies can be used for recovery:

[0089] 1) Model parameter-level reload strategy: Reload model weights or model parameters without destroying the container. 2) Model runtime mode switching strategy: Switch the model instance to low-precision, simplified computation, or standby model mode. 3) Model instance reconstruction strategy: Rebuild the model runtime environment on the target node while maintaining the model instance identifier. 4) Model instance restart strategy: Restart the container corresponding to the model instance.

[0090] If the target model instance is in a running state or a degraded running state after the preset recovery strategy is executed, then the lifecycle of the target model instance is updated and external services are restored.

[0091] The recovery result is determined. If the model instance is restored to the running state (S2) or the degraded running state (S2-1), the lifecycle state is updated and external services are restored. If the recovery fails, the model instance state is switched to the stopped state (S4).

[0092] Otherwise, the target model instance is switched to a stopped state; relevant information about the target model instance is stored; and the scheduling strategy, runtime resource allocation strategy, and lifecycle state transition rules of similar models of the target model instance are adjusted according to the relevant information.

[0093] After a model instance enters a stopped state, a model instance recycling process is executed. Before recycling, relevant information such as the model's running status, exception context, and key operational data are saved for subsequent analysis or model optimization. The exception type, exception confidence score, recovery strategy, and recovery results of the target model instance can be fed back to model instance management and runtime resource management for dynamic adjustment of subsequent model instance scheduling strategies, runtime resource allocation strategies, and lifecycle state migration rules. Specifically, for model instances with frequent historical anomalies or a high probability of recovery failure, redundant computing resources are prioritized during subsequent startup or reconstruction processes, the model warm-up period is extended, or the instance directly enters a degraded running state to reduce the risk of recurrence of anomalies.

[0094] Through the aforementioned closed-loop mechanism for anomaly handling, recovery, and feedback based on model semantics, the model instance lifecycle state machine possesses the ability to adaptively evolve based on historical operational behavior. This model instance-level autonomous recovery and adaptive control capability, distinct from current container restart methods, significantly improves the stability, reliability, and continuous operation of model services in industrial scenarios.

[0095] The above implementation method enables standardized recovery of target model instances based on abnormal states and reclaims them when recovery fails, achieving automated management of target model instances. Furthermore, adjustments are made based on relevant information about the target model instances to dynamically reflect the resource and operational requirements of the model instances, thereby improving the accuracy of model instance scheduling and resource allocation.

[0096] The model instance monitoring method for a container platform provided in this invention generates a behavioral feature vector based on the basic operating environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance when the target model instance is in a preset state. The method then determines the confidence level corresponding to multiple operating states of the target model instance based on the behavioral feature vector and a comprehensive judgment model. Finally, it determines the actual operating state of the target model instance based on the confidence levels corresponding to the multiple operating states. If the actual operating state is an abnormal state, the target model instance is restored or reclaimed. Compared to the current inability to monitor model instances in containers, the model instance monitoring method for a container platform provided in this invention can generate a behavioral feature vector based on the basic operating environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance in a preset state. The behavioral feature vector comprehensively represents the operating condition of the target model instance in vector form. The method determines the confidence level corresponding to multiple operating states of the target model instance based on the behavioral feature vector and a comprehensive judgment model, and determines the actual operating state of the target model instance based on the confidence levels corresponding to the multiple operating states. This achieves accurate determination of the actual operating state of the target model instance based on the behavioral feature vector, thus completing the monitoring of the operating condition of model instances in containers. Based on this, if the actual operating state is abnormal, the target model instance will be restored or recycled in a timely manner, thereby improving the reliability of the model instance and increasing resource utilization.

[0097] Figure 3 This is a schematic diagram illustrating a model instance monitoring method for a container platform provided in an embodiment of the present invention. As a further explanation of the above-described implementation, it includes:

[0098] Step S201: Pre-acquire a training set, which includes training behavior feature vectors and running status labels; use the training set to train the initial model using a preset training method to obtain a comprehensive judgment model.

[0099] Step S202: Configure the state machine of the target model instance. The state machine includes initialization state, startup state, model preheating state, running state, degenerate running state, abnormal state, stopped state, and recycled state.

[0100] Step S203: When the target model instance in the container is in a running state or a degraded running state, obtain the basic running environment state, heartbeat level information, inference behavior characteristics and inference result characteristics of the target model instance.

[0101] Step S204: Determine whether the heartbeat level information and reasoning behavior characteristics are abnormal according to preset rules. The preset rules include: determining whether it is abnormal based on the reasoning response time and timeout threshold; determining whether it is abnormal based on the reasoning success rate and success rate threshold; and determining whether it is abnormal based on whether the heartbeat level information is interrupted.

[0102] Step S205: Digitally map the basic operating environment state, heartbeat level information, reasoning behavior features and reasoning result features of the target model instance to obtain corresponding single-dimensional vectors; combine multiple single-dimensional vectors to obtain a behavior feature vector.

[0103] Step S206: Input the feature vector into the comprehensive judgment model. The comprehensive judgment model is used to perform state machine classification based on the feature input vector and output the confidence level of the target model instance in the abnormal state and the degenerate running state, respectively.

[0104] Step S207: The confidence scores of abnormal states and degraded operating states are weighted and summed according to preset weighting parameters to obtain the actual confidence score; the actual operating state of the target model instance is determined according to the actual confidence score and the preset confidence threshold.

[0105] If the target model instance is in a running state or a degraded running state, then return to step S203.

[0106] Step S208: If the target model instance is in an abnormal state, it shall be restored according to a preset recovery strategy. The preset recovery strategy includes: model parameter level reload strategy, model running mode switching strategy, model instance reconstruction strategy, or model instance restart strategy.

[0107] Step S209: If the target model instance is in a running state or a degraded running state after the preset recovery strategy is executed, then update the lifecycle of the target model instance and restore external services;

[0108] Step S210: Otherwise, switch the target model instance to a stopped state; store the relevant information of the target model instance; adjust the scheduling strategy, runtime resource allocation strategy, and lifecycle state transition rules of similar models of the target model instance according to the relevant information.

[0109] In this embodiment of the invention, the comprehensive judgment model is a state discrimination model oriented towards the behavioral characteristics of model instances. Its judgment object is the operational behavior and output logic of the model instance itself, rather than solely based on the resource state of the container in which the model instance resides. The comprehensive judgment model can be implemented using a combination of rule-based models and machine learning models. This embodiment of the invention can accurately identify the abnormal state of a model instance when the container is running normally but the model instance's logic or inference behavior is abnormal, thereby accurately identifying the true operational state of the model instance and achieving model instance-level operational state awareness.

[0110] Figure 4 This is a schematic diagram of a model instance monitoring device for a container platform provided in an embodiment of the present invention. This embodiment is applicable to situations where model instances in containers are monitored and managed. This device can be executed by a model instance monitoring device for the container platform, which can be implemented in hardware and / or software. This model instance monitoring device for the container platform can be configured in electronic devices such as personal computers and servers. Figure 4 As shown, the device includes: a behavior feature vector generation module 31, an operation status prediction module 32, an actual operation status determination module 33, and an exception handling module 34.

[0111] The behavior feature vector generation module 31 is used to generate a behavior feature vector based on the basic operating environment state, heartbeat level information, reasoning behavior features and reasoning result features of the target model instance when the target model instance in the container is in a preset state.

[0112] The running state prediction module 32 is used to determine the confidence level corresponding to multiple running states of the target model instance based on the behavior feature vector and the comprehensive judgment model;

[0113] The actual running state determination module 33 is used to determine the actual running state of the target model instance based on the confidence levels corresponding to the multiple running states.

[0114] The exception handling module 34 is used to restore or reclaim the target model instance if the actual running state is an abnormal state.

[0115] Based on the above embodiments, optionally, a state machine configuration module is also included, which is used to configure the state machine of the target model instance before generating a behavior feature vector based on the basic running environment state, heartbeat level information, inference behavior characteristics and inference result characteristics of the target model instance when the target model instance in the container is in a preset state. The state machine includes an initialization state, a startup state, a model preheating state, a running state, a degraded running state, an abnormal state, a stopped state and a recycled state.

[0116] Accordingly, the behavior feature vector generation module 31 is used for:

[0117] When the target model instance in the container is in a running state or a degraded running state, a behavior feature vector is generated based on the basic running environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance.

[0118] Based on the above embodiments, optionally, the behavior feature vector generation module 31 is used for:

[0119] Obtain the basic runtime environment status of the target model instance, which includes: the running status of the container, the CPU usage information of the container, the memory usage information of the container, the GPU usage information of the container, or the network connection status.

[0120] Obtain the heartbeat level information of the target model instance, which includes the following information: the version identifier of the target model instance, the loading status of the target model instance, the current number of service requests of the target model instance, the most recent inference response time of the target model instance, or the abnormal label of the target model instance.

[0121] The inference behavior characteristics of the target model instance are obtained, including: inference response time and its fluctuation characteristics, inference request queue length, inference success rate, and inference request throughput per unit time.

[0122] Obtain the inference result features of the target model instance, including: the distribution characteristics of the inference output results, the trend of inference result changes, the stability index of the inference results, or the degree of deviation of the inference results from historical working condition data;

[0123] The basic operating environment state, heartbeat level information, reasoning behavior characteristics, and reasoning result characteristics of the target model instance are digitally mapped to obtain corresponding single-dimensional vectors; multiple single-dimensional vectors are combined to obtain a behavior feature vector.

[0124] Based on the above embodiments, optionally, a rule recognition module is also included, used to determine the anomalies of heartbeat-level information and inference behavior features according to preset rules before determining the confidence levels corresponding to multiple running states of the target model instance based on the behavior feature vector and the comprehensive judgment model. The preset rules include: determining whether it is abnormal based on inference response time and timeout threshold; determining whether it is abnormal based on inference success rate and success rate threshold; and determining whether it is abnormal based on whether the heartbeat-level information is interrupted.

[0125] Based on the above embodiments, optionally, a model training module is also included. The model training module is used to: pre-acquire a training set, the training set including training behavior feature vectors and running status labels; and use the training set to train an initial model using a preset training device to obtain a comprehensive judgment model.

[0126] The running state prediction module 32 is used to: input the feature vector into the comprehensive judgment model when in use, and the comprehensive judgment model is used to perform state machine classification based on the feature input vector and output the confidence of the target model instance being in an abnormal state and a degenerate running state, respectively.

[0127] Based on the above embodiments, optionally, the actual operating status determination module 33 is used for:

[0128] The confidence levels of abnormal and degraded operating states are weighted and summed according to preset weighting parameters to obtain the actual confidence level.

[0129] The actual running state of the target model instance is determined based on the actual confidence level and the preset confidence threshold.

[0130] Based on the above embodiments, optionally, the exception handling module 34 is used for:

[0131] If the target model instance is in an abnormal state, it is restored according to a preset recovery strategy. The preset recovery strategy includes: model parameter level reload strategy, model running mode switching strategy, model instance reconstruction strategy, or model instance restart strategy.

[0132] If the target model instance is in a running state or a degraded running state after the preset recovery strategy is executed, then the lifecycle of the target model instance is updated and external services are restored.

[0133] Otherwise, the target model instance is switched to a stopped state; relevant information about the target model instance is stored; and the scheduling strategy, runtime resource allocation strategy, and lifecycle state transition rules of similar models of the target model instance are adjusted according to the relevant information.

[0134] The container platform model instance monitoring device provided in this embodiment of the invention includes a behavior feature vector generation module 31, used to generate a behavior feature vector based on the basic operating environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance when the target model instance in the container is in a preset state; an operating state prediction module 32, used to determine the confidence level corresponding to multiple operating states of the target model instance based on the behavior feature vector and a comprehensive judgment model; an actual operating state determination module 33, used to determine the actual operating state of the target model instance based on the confidence level corresponding to the multiple operating states; and an anomaly handling module 34, used to restore or reclaim the target model instance if the actual operating state is an abnormal state. Compared to the current inability to monitor model instances within containers, the model instance monitoring device for a container platform provided in this embodiment of the invention can generate behavioral feature vectors based on the basic operating environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of a target model instance in a preset state. These behavioral feature vectors comprehensively represent the operating status of the target model instance in vector form. Based on the behavioral feature vectors and a comprehensive judgment model, the confidence levels corresponding to multiple operating states of the target model instance are determined. The actual operating state of the target model instance is then determined based on the confidence levels corresponding to these multiple operating states. This achieves accurate determination of the actual operating state of the target model instance based on the behavioral feature vectors, completing the monitoring of the operating status of model instances within the container. Furthermore, if the actual operating state is abnormal, the target model instance is restored or reclaimed, enabling timely restoration and reclamation of abnormal target model instances, improving the reliability of model instances, and increasing resource utilization.

[0135] The container platform model instance monitoring device provided in this embodiment of the invention can execute the container platform model instance monitoring method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0137] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the model instance monitoring method of a container platform.

[0139] In some embodiments, the model instance monitoring method for the container platform can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the model instance monitoring method for the container platform described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the model instance monitoring method for the container platform by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs for implementing the model instance monitoring method of the container platform of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] The invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a model instance monitoring method for a container platform, characterized in that it includes:

[0143] When the target model instance in the container is in a preset state, a behavior feature vector is generated based on the basic runtime environment state, heartbeat level information, reasoning behavior characteristics, and reasoning result characteristics of the target model instance.

[0144] The confidence levels corresponding to multiple running states of the target model instance are determined based on the behavioral feature vector and the comprehensive judgment model.

[0145] The actual running state of the target model instance is determined based on the confidence levels corresponding to the multiple running states.

[0146] If the actual operating state is abnormal, the target model instance will be restored or recycled.

[0147] Based on the above embodiments, optionally, before generating the behavior feature vector according to the basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance when the target model instance in the container is in a preset state, the method further includes:

[0148] Configure the state machine of the target model instance, the state machine includes initialization state, startup state, model warm-up state, running state, degenerate running state, abnormal state, stopped state and recycled state;

[0149] Accordingly, when the target model instance in the container is in a preset state, a behavioral feature vector is generated based on the target model instance's basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics, including:

[0150] When the target model instance in the container is in a running state or a degraded running state, a behavior feature vector is generated based on the basic running environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance.

[0151] Based on the above embodiments, optionally, a behavioral feature vector is generated according to the basic operating environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance, including:

[0152] Obtain the basic runtime environment status of the target model instance, which includes: the running status of the container, the CPU usage information of the container, the memory usage information of the container, the GPU usage information of the container, or the network connection status.

[0153] Obtain the heartbeat level information of the target model instance, which includes the following information: the version identifier of the target model instance, the loading status of the target model instance, the current number of service requests of the target model instance, the most recent inference response time of the target model instance, or the abnormal label of the target model instance.

[0154] The inference behavior characteristics of the target model instance are obtained, including: inference response time and its fluctuation characteristics, inference request queue length, inference success rate, and inference request throughput per unit time.

[0155] Obtain the inference result features of the target model instance, including: the distribution characteristics of the inference output results, the trend of inference result changes, the stability index of the inference results, or the degree of deviation of the inference results from historical working condition data;

[0156] The basic operating environment state, heartbeat level information, reasoning behavior characteristics, and reasoning result characteristics of the target model instance are digitally mapped to obtain corresponding single-dimensional vectors; multiple single-dimensional vectors are combined to obtain a behavior feature vector.

[0157] Based on the above embodiments, optionally, before determining the confidence levels corresponding to multiple running states of the target model instance based on the behavioral feature vector and the comprehensive judgment model, the method further includes:

[0158] Anomalies are determined based on preset rules for heartbeat-level information and inference behavior characteristics. These preset rules include: determining whether anomalies occur based on inference response time and timeout threshold; determining whether anomalies occur based on inference success rate and success rate threshold; and determining whether anomalies occur based on whether heartbeat-level information is interrupted.

[0159] Based on the above embodiments, optionally, the confidence levels corresponding to multiple running states of the target model instance are determined according to the behavioral feature vector and the comprehensive judgment model, including:

[0160] A training set is pre-acquired, which includes training behavior feature vectors and running status labels; using the training set, an initial model is trained using a preset training method to obtain a comprehensive judgment model;

[0161] In use, the feature vector is input into the comprehensive judgment model, which is used to perform state machine classification based on the feature input vector and output the confidence level of the target model instance in the abnormal state and the degenerate running state, respectively.

[0162] Based on the above embodiments, optionally, determining the actual running state of the target model instance according to the confidence levels corresponding to the multiple running states includes:

[0163] The confidence levels of abnormal and degraded operating states are weighted and summed according to preset weighting parameters to obtain the actual confidence level.

[0164] The actual running state of the target model instance is determined based on the actual confidence level and the preset confidence threshold.

[0165] Based on the above embodiments, optionally, if the actual operating state is an abnormal state, the target model instance is restored or recycled, including:

[0166] If the target model instance is in an abnormal state, it is restored according to a preset recovery strategy. The preset recovery strategy includes: model parameter level reload strategy, model running mode switching strategy, model instance reconstruction strategy, or model instance restart strategy.

[0167] If the target model instance is in a running state or a degraded running state after the preset recovery strategy is executed, then the lifecycle of the target model instance is updated and external services are restored.

[0168] Otherwise, the target model instance is switched to a stopped state; relevant information about the target model instance is stored; and the scheduling strategy, runtime resource allocation strategy, and lifecycle state transition rules of similar models of the target model instance are adjusted according to the relevant information.

[0169] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0171] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0172] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0173] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0174] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for monitoring model instances on a container platform, characterized in that, include: When the target model instance in the container is in a preset state, a behavior feature vector is generated based on the basic runtime environment state, heartbeat level information, reasoning behavior characteristics, and reasoning result characteristics of the target model instance. The confidence levels corresponding to multiple running states of the target model instance are determined based on the behavioral feature vector and the comprehensive judgment model. The actual running state of the target model instance is determined based on the confidence levels corresponding to the multiple running states. If the actual operating state is abnormal, the target model instance will be restored or recycled.

2. The method according to claim 1, characterized in that, Before generating the behavior feature vector based on the target model instance's basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics when the target model instance is in a preset state within the container, the process further includes: Configure the state machine of the target model instance, the state machine includes initialization state, startup state, model warm-up state, running state, degenerate running state, abnormal state, stopped state and recycled state; Accordingly, when the target model instance in the container is in a preset state, a behavioral feature vector is generated based on the target model instance's basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics, including: When the target model instance in the container is in a running state or a degraded running state, a behavior feature vector is generated based on the basic running environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance.

3. The method according to claim 2, characterized in that, Based on the basic runtime environment state, heartbeat level information, inference behavior characteristics, and inference result characteristics of the target model instance, a behavioral feature vector is generated, including: Obtain the basic runtime environment status of the target model instance, which includes: the running status of the container, the CPU usage information of the container, the memory usage information of the container, the GPU usage information of the container, or the network connection status. Obtain the heartbeat level information of the target model instance, which includes the following information: the version identifier of the target model instance, the loading status of the target model instance, the current number of service requests of the target model instance, the most recent inference response time of the target model instance, or the abnormal label of the target model instance. The inference behavior characteristics of the target model instance are obtained, including: inference response time and its fluctuation characteristics, inference request queue length, inference success rate, and inference request throughput per unit time. Obtain the inference result features of the target model instance, including: the distribution characteristics of the inference output results, the trend of inference result changes, the stability index of the inference results, or the degree of deviation of the inference results from historical working condition data; The basic operating environment state, heartbeat level information, reasoning behavior characteristics, and reasoning result characteristics of the target model instance are digitally mapped to obtain corresponding single-dimensional vectors; multiple single-dimensional vectors are combined to obtain a behavior feature vector.

4. The method according to claim 3, characterized in that, Before determining the confidence levels corresponding to multiple running states of the target model instance based on the behavioral feature vector and the comprehensive judgment model, the method further includes: Anomalies are determined based on preset rules for heartbeat-level information and inference behavior characteristics. These preset rules include: determining whether anomalies occur based on inference response time and timeout threshold; determining whether anomalies occur based on inference success rate and success rate threshold; and determining whether anomalies occur based on whether heartbeat-level information is interrupted.

5. The method according to claim 4, characterized in that, The confidence levels corresponding to multiple running states of the target model instance are determined based on the behavioral feature vector and the comprehensive judgment model, including: A training set is pre-acquired, which includes training behavior feature vectors and running status labels; using the training set, an initial model is trained using a preset training method to obtain a comprehensive judgment model; In use, the feature vector is input into the comprehensive judgment model, which is used to perform state machine classification based on the feature input vector and output the confidence level of the target model instance in the abnormal state and the degenerate running state, respectively.

6. The method according to claim 5, characterized in that, The actual running state of the target model instance is determined based on the confidence levels corresponding to the multiple running states, including: The confidence levels of abnormal and degraded operating states are weighted and summed according to preset weighting parameters to obtain the actual confidence level. The actual running state of the target model instance is determined based on the actual confidence level and the preset confidence threshold.

7. The method according to claim 2, characterized in that, If the actual operating state is abnormal, the target model instance is restored or recycled, including: If the target model instance is in an abnormal state, it is restored according to a preset recovery strategy. The preset recovery strategy includes: model parameter level reload strategy, model running mode switching strategy, model instance reconstruction strategy, or model instance restart strategy. If the target model instance is in a running state or a degraded running state after the preset recovery strategy is executed, then the lifecycle of the target model instance is updated and external services are restored. Otherwise, the target model instance is switched to a stopped state; relevant information about the target model instance is stored; and the scheduling strategy, runtime resource allocation strategy, and lifecycle state transition rules of similar models of the target model instance are adjusted according to the relevant information.

8. A model instance monitoring device for a container platform, characterized in that, include: The behavior feature vector generation module is used to generate a behavior feature vector based on the basic runtime environment state, heartbeat level information, inference behavior features and inference result features of the target model instance when the target model instance in the container is in a preset state. The running state prediction module is used to determine the confidence level corresponding to multiple running states of the target model instance based on the behavioral feature vector and the comprehensive judgment model; The actual running status determination module is used to determine the actual running status of the target model instance based on the confidence levels corresponding to the multiple running states. An exception handling module is used to restore or reclaim the target model instance if the actual running state is an abnormal state.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the model instance monitoring method of the container platform according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the model instance monitoring method of the container platform according to any one of claims 1-7.

Images