Automatic monitoring operation and maintenance method for AI visual analysis service and related device

Abstract

The invention provides an AI visual analysis service automatic monitoring operation and maintenance method and a related device, and the method comprises the steps: providing an interface for forcibly starting a fault checking process, enabling an operation and maintenance worker or a technician to forcibly start an automatic fault checking process through calling the interface when the automatic inspection does not give an alarm, and enabling the operation and maintenance worker or the technician to carry out the operation and maintenance of the AI visual analysis service. And a troubleshooting result is returned in a synchronous or asynchronous manner. Therefore, operation and maintenance or research and development personnel do not need to manually log in a system environment and manually confirm one by one according to an automatic inspection flow, so that the fault processing efficiency and the fault processing accuracy can be improved. According to whether the AI middle station generates the target alarm data or not, the subscription information and the push log of the target AI visual analysis service are selected to carry out fault positioning, or the task details of the AI middle station, the target equipment state and the target algorithm state corresponding to the target AI visual analysis service are selected to carry out fault positioning, so that an accurate fault positioning result can be obtained, and the fault positioning accuracy is improved. And the fault processing accuracy can be further improved.

Description

Technical Field

[0001] The present application relates to the field of monitoring and operation and maintenance technology, and in particular to an automated monitoring and operation and maintenance method and related devices for AI visual analysis services. Background Art

[0002] At present, AI vision technology has been widely used in all walks of life, and a large number of AI video analysis systems that integrate computing power, algorithms, and platforms have emerged to uniformly manage and automatically train AI vision algorithms. These AI video analysis systems provide more than 500 million AI vision analysis computing services per day, so their system stability and reliability are related to the service quality of AI vision analysis services.

[0003] In order to improve system stability and reliability, in addition to continuously optimizing system architecture and system performance, reducing the frequency of failures, narrowing the scope of failure impact and reducing failure recovery time are one of the important means to improve system stability and reliability. This requires that system failures can be discovered and handled in a timely manner during the system monitoring and operation process.

[0004] At present, some existing implementation methods rely on professional technicians to perform fault analysis, troubleshooting and fault handling. Take the AI ​​vision atomic capability response timeout problem with many complaints and the third-party platform not receiving the algorithm analysis test results for a long time after subscribing to the AI ​​vision atomic capability as an example. After the above problems occur, the maintenance and management of the system often rely on professional technicians to perform fault analysis and analyze whether the problem is caused by factors such as interface service call response timeout, network failure causing AI vision algorithm to parse video stream abnormally, and AI vision algorithm service itself. However, considering the wide variety of fault causes, if manual troubleshooting and fault handling are used, it will lead to high operation and maintenance costs, low efficiency and easy errors. At the same time, with the continuous expansion of business scale and the continuous increase in system complexity, this puts higher and higher requirements on system stability and reliability, making it difficult for manual monitoring and operation and maintenance methods to meet the actual needs of early detection, accurate positioning and fast delivery.

[0005] To solve the above problems, the prior art proposes an automated operation and maintenance method for centralized monitoring of products. The method realizes centralized monitoring and automated operation and maintenance of products by setting product detection points, uniformly processing monitoring data, visually displaying processing results on the monitoring platform, intelligently analyzing faults and sending them to the automated operation and maintenance terminal. This method can improve the accuracy and timeliness of monitoring, reduce labor costs, improve operation and maintenance efficiency, and realize intelligent analysis and alarm of product faults. This method can help R&D personnel better manage and monitor products and improve product stability and reliability.

[0006] However, existing automated monitoring and operation solutions still lack efficiency and accuracy in troubleshooting. Summary of the invention

[0007] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defects of low fault handling efficiency and low fault handling accuracy in the prior art.

[0008] In a first aspect, an embodiment of the present application provides an automated monitoring and operation method for an AI visual analysis service, including:

[0009] In response to the troubleshooting interface being called or the automatic inspection result being abnormal, determining whether the AI ​​middle station generates target alarm data for the target AI visual analysis service; wherein the target AI visual analysis service is the AI ​​visual analysis service that needs troubleshooting;

[0010] If the AI ​​middle station generates the target alarm data, the fault location is performed according to the subscription information and push log of the target AI visual analysis service, and the fault location result is obtained;

[0011] If the AI ​​middle station does not generate the target alarm data, the fault location is performed according to the task details of the AI ​​middle station, the target device status and the target algorithm status corresponding to the target AI visual analysis service, and the fault location result is obtained; wherein the target device status is the status of the device used to run the target AI visual analysis service;

[0012] Generate and push a fault troubleshooting result based on the fault location result.

[0013] In a second aspect, an embodiment of the present application provides an automated monitoring and operation device for an AI visual analysis service, including:

[0014] A judgment module, used to judge whether the AI ​​middle station generates target alarm data for a target AI visual analysis service in response to the fault troubleshooting interface being called or the automatic inspection result being abnormal; wherein the target AI visual analysis service is an AI visual analysis service that needs to be troubleshooted;

[0015] The first fault location module is used to locate the fault according to the subscription information and push log of the target AI visual analysis service if the AI ​​middle station generates the target alarm data, and obtain the fault location result;

[0016] A second fault location module is used to locate the fault according to the task details of the AI ​​middle station, the target device status and the target algorithm status corresponding to the target AI visual analysis service if the AI ​​middle station does not generate the target alarm data, and obtain the fault location result; wherein the target device status is the status of the device used to run the target AI visual analysis service;

[0017] The push module is used to generate and push the fault troubleshooting result according to the fault location result.

[0018] In an automated monitoring and operation method and related device for an AI visual analysis service provided in some embodiments of the present application, an interface for forcibly starting a troubleshooting process is provided. When the automated inspection does not generate an alarm, the operation and maintenance personnel or technical personnel can call the interface to forcibly start the automated troubleshooting process for analyzing the fault, and can receive the troubleshooting results returned in a synchronous or asynchronous manner. In this way, there is no need for the operation and maintenance or R&D personnel to manually log in to the system environment and manually confirm one by one according to the automated inspection process, thereby improving the efficiency and accuracy of fault handling.

[0019] At the same time, this application can use the subscription information and push logs of the target AI visual analysis service to locate the fault based on whether the AI ​​middle station generates target alarm data, or use the task details of the AI ​​middle station, the target device status, and the target algorithm status corresponding to the target AI visual analysis service to locate the fault, thereby obtaining accurate fault location results and further improving the accuracy of fault handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 This is one of the flowcharts of the automated monitoring and operation method of the AI ​​visual analysis service in one embodiment;

[0022] Figure 2 One of the schematic diagrams of the troubleshooting process in one embodiment;

[0023] Figure 3A One of the schematic diagrams of monitoring indicators in one embodiment;

[0024] Figure 3B This is a second schematic diagram of monitoring indicators in one embodiment;

[0025] Figure 4 This is a second schematic diagram of a troubleshooting process in an embodiment;

[0026] Figure 5 In one embodiment, a visual troubleshooting flowchart is shown.

[0027] Figure 6 1 is a schematic diagram of the structure of an automated monitoring and operation device for an AI visual analysis service in one embodiment;

[0028] Figure 7 FIG. 1 is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The following is an explanation of the terms involved in this application:

[0031] AI (Artificial Intelligence) Visual Analysis: Use artificial intelligence technology to automatically analyze and understand images or videos, extract useful information, and achieve functions such as target detection, tracking, and recognition.

[0032] Low-code platform: A software development platform that allows users to quickly build applications through a graphical interface and a small amount of code, lowering the development threshold and cycle, improving development efficiency, and quickly producing applications, thereby speeding up product time to market.

[0033] Automated and precise monitoring: Through preset alarm rules, standard fault handling processes and algorithms, the system or business is automatically monitored in real time and faults are self-healed to ensure timely detection and handling of abnormal situations and achieve precise operation and maintenance.

[0034] As mentioned in the background technology, the existing technology still has deficiencies in fault handling efficiency and accuracy. The inventors have found that the cause of this problem is that when the automatic inspection of the business does not generate an alarm, if the operation personnel or front-end personnel receive complaints from customers / applications, and believe that the specified task should generate an alarm but does not actually generate an alarm, the operation and maintenance or R&D personnel are often required to manually log in to the system environment and manually confirm one by one according to the automatic inspection process. This greatly affects the efficiency and accuracy of fault handling.

[0035] To solve the above problems, the present application provides an interface for directly querying a specified fault. By calling the interface, the user can forcefully start the troubleshooting process for analyzing the fault and receive the troubleshooting results returned in a synchronous or asynchronous manner.

[0036] In addition to the technical problems mentioned above, the existing technology has at least the following four problems:

[0037] (1) The prior art first sets up product monitoring points, and centrally monitors the product monitoring points based on preset monitoring rules, and obtains monitoring data. In this process, the prior art mainly monitors the indicator data of service quality indicators (such as health status indicators of middleware, database, virtualization, network equipment, server, host hardware, etc.) through product monitoring points, and ignores the monitoring of business indicators. However, normal service quality indicators do not mean that business indicators are also normal. For example, when a network failure occurs from the client to the server, service quality indicators such as the success rate and delay value of the server are all normal values, but for the client, it cannot subscribe to the AI ​​vision service, cannot call the AI ​​vision service for video analysis, and cannot receive the algorithm results pushed by the AI ​​vision service.

[0038] Although there are many alarm configuration strategies for cause-based indicators, such as machine downtime alarm, high CPU utilization alarm, high memory usage alarm, etc., the monitoring of business indicators is ignored, so it is impossible to clearly define the impact of cause-based indicators on the business level. This lack of specificity leads to a high rate of missed fault alarms, and it is impossible to ensure that there will be no alarms for every fault.

[0039] (2) In the existing technology, product monitoring points rely on a wide variety of monitoring and collection tools, which not only brings inconsistency in the operating experience, but also increases the learning cost of technical personnel. In addition, the configuration process of monitoring alarm strategies is complicated and cumbersome, and the maintenance work is heavy. These require personnel with professional skills to undertake, which increases the cost of monitoring operation and maintenance. The high cost of monitoring operation and maintenance and delivery makes it difficult for the entire system to flexibly adapt to rapidly changing business needs.

[0040] (3) Data silos are prominent, and operation and maintenance personnel need to check each independent monitoring data source one by one. This process is not only costly and inefficient in correlation analysis, but also greatly increases the complexity and time-consuming nature of problem troubleshooting. This decentralized data management method often leads to untimely exception handling, making it difficult to effectively ensure the stable operation of the platform, and even more difficult to achieve the goals of early fault detection, accurate positioning, and rapid loss prevention.

[0041] (4) The operation and maintenance team may face hundreds or even more alarms every day. In many cases, only a few of them are valid fault alarms that need to be handled immediately. However, these key alarms are often overwhelmed by a large number of false alarms, making it difficult to identify and handle them in a timely manner. What is more difficult is that different roles such as operations, R&D, and customers have different concerns about system failures, making it difficult for each position to respond to and handle real problems in the first place.

[0042] In addition, although the existing alarm system can basically realize the fault notification function and describe the cause and severity of the fault in a preset way, it lacks in-depth analysis and it is difficult to provide key information such as the scope of fault impact, root cause and repair suggestions. This makes it difficult for operation and maintenance and R&D personnel to judge the effectiveness of automatic repair measures. They often need to manually log in to the system environment to confirm the repair effect one by one, which greatly affects the efficiency and accuracy of fault handling.

[0043] To solve the above problems, in some embodiments, this application integrates service quality target monitoring indicators and AI visual business monitoring indicators in the automated monitoring and operation of AI video services, builds a comprehensive business monitoring system, realizes flexible configuration and customized monitoring, unified monitoring probes to simplify management, intelligent detection and early warning, efficient fault handling and location, in-depth analysis and automatic repair, etc., which significantly improves the operation and maintenance efficiency and system stability of AI visual services and has strong practicality. The automated monitoring and operation and maintenance solution for AI visual analysis services provided in this application includes at least one of the following five aspects:

[0044] (1) Combine the indicators for monitoring service quality goals with the indicators for AI visual analysis business, attach importance to the construction and monitoring of the business indicator system, enumerate the result indicators and fault categories corresponding to the core business functions, make good monitoring rules for the result indicators, and build a more comprehensive business monitoring indicator system. This system is conducive to the efficient operation of AI visual business and the continuous optimization of products.

[0045] (2) Integrate all detection and collection tools related to AI visual business analysis and encapsulate them into a unified monitoring probe collector, which can fully cover all monitoring targets, realize centralized management, simplify the configuration process, and allow users to directly specify the capture target details, capture rules, etc. in the configuration of the monitoring probe collector, thereby automatically completing the collection, cleaning and classification of monitoring data. In addition, by formulating a standardized template for the configuration file, not only is the problem of different configuration file formats and uneven operating experience between different plug-ins eliminated, but it also significantly lowers the learning threshold of the technical team, promotes the rapid deployment and delivery of the project, and better adapts to the rapid changes in market and business needs. The unified monitoring probe collector can not only integrate third-party plug-ins, but also expand custom monitoring targets, obtain data from the business database forms, and generate key business indicators based on preset algorithms, such as the use of business core function service calls and AI visual alarm push volume. According to the pre-established data specifications, these data are converted into a unified format, and a series relationship is established between these data, which are summarized into business indicators. Intelligent anomaly detection is performed on the indicators in combination with business needs. Abnormal trends in business indicators are discovered and alarms are issued in a timely manner, truly turning passivity into initiative and discovering fault problems before complaints are made.

[0046] (3) The low-code platform can be used to quickly build and change the notification method, alarm strategy and visual monitoring template of the alarm, simplify the configuration of monitoring rules, lower the technical threshold and reduce the maintenance cost. When a business is about to fail or there is a potential failure, not only can the first event be actively discovered by the monitoring and an early warning can be issued, but also an in-depth analysis can be automatically performed, and the method of eliminating the potential failure can be accurately described through the early warning notification. For the active discovery of the fault, not only can the preliminary location be automatically performed, and the scope of the fault point can be accurately narrowed and described through the alarm notification, but also an in-depth analysis can be automatically performed, and the cause of the fault and the method of repairing the fault can be accurately and detailedly described through the alarm. While transmitting the data to be transmitted to the alarm terminal in the form of an alarm, the fault can be automatically repaired if conditions permit. In addition, the reasonable configuration of the alarm noise reduction and alarm suppression strategies can intuitively and efficiently conduct closed-loop tracking of troubleshooting of these alarm problems.

[0047] (4) According to different user roles (maintenance, operation or R&D personnel), users can subscribe to the monitoring rules that are relevant to them, subscribe to unified and standardized monitoring rules, and redefine the alarm level, notification method and callback interface, thereby improving the alarm response speed. This helps to better adapt to the ever-changing market environment and accelerate the promotion of AI visual analysis services.

[0048] (5) Monitoring and early warning are combined with in-depth analysis. The alarm notification in step 3 not only automatically provides methods to eliminate potential faults, but also includes fault location, cause analysis, and repair guidance. Since the task of in-depth analysis of business faults is essentially a process of automatic inspection and subsequent detailed analysis, there will inevitably be a certain time delay. Users can force the start of a customized in-depth analysis of business faults troubleshooting process, and return the results of the specified troubleshooting (for example, including fault location, cause analysis, and repair guidance) in a synchronous or asynchronous manner.

[0049] The following is an explanation of the automated monitoring and operation method for the AI ​​visual analysis service provided by this application.

[0050] In some embodiments, Figure 1 As shown, the present application provides an automated monitoring and operation method for AI visual analysis services, comprising the following steps:

[0051] S102: In response to the troubleshooting interface being called or the automatic inspection result being abnormal, determining whether the AI ​​middle station generates target alarm data for the target AI visual analysis service; wherein the target AI visual analysis service is the AI ​​visual analysis service that needs troubleshooting;

[0052] S104: If the AI ​​middle station generates target alarm data, the fault is located according to the subscription information and push log of the target AI visual analysis service, and the fault location result is obtained;

[0053] S106: If the AI ​​middle station does not generate target alarm data, the fault location is performed according to the task details of the AI ​​middle station, the target device status, and the target algorithm status corresponding to the target AI visual analysis service, and the fault location result is obtained; wherein the target device status is the status of the device used to run the target AI visual analysis service;

[0054] S108: Generate and push a fault troubleshooting result based on the fault location result.

[0055] Among them, the troubleshooting interface is an interface used to forcibly start the troubleshooting process. Operation and maintenance personnel or technical personnel can start the troubleshooting process for the target AI visual analysis business by calling the troubleshooting interface.

[0056] This application provides an automatic inspection function and an interface for forcibly starting a troubleshooting process. When the automatic inspection result is abnormal, the troubleshooting process can be started to troubleshoot the target AI visual analysis business. When the troubleshooting interface is called, this application can also start the troubleshooting process. It should be noted that when the troubleshooting interface is called, regardless of whether the automated inspection structure generates an alarm, this application must force the execution of the troubleshooting process to deeply analyze the business failure and return the troubleshooting results.

[0057] For example, when the automatic inspection does not generate an alarm, but the operator or front-end personnel receives a specified task reported by the customer or application side that should generate an alarm but does not actually generate an alarm, the fault troubleshooting interface can be called to force the start of a customized in-depth analysis of the business fault troubleshooting process and return the troubleshooting results. This method is mainly for items with a long time interval for automatic inspections, or a long execution frequency and execution time set in the process of intelligent noise reduction, because it may cause the inspection results to fail to transmit the detected faults to the alarm terminal in the first time. Therefore, when the operator or front-end personnel receive complaints from customers or applications, after adding a specified task that requires in-depth analysis of the business fault location results, the custom monitoring business process will force the start of the in-depth analysis of the business fault troubleshooting logic process, and return the specified troubleshooting results synchronously or asynchronously.

[0058] In the process of executing the troubleshooting process, this application can first check the alarm data of the AI ​​middle station to determine whether the AI ​​middle station has generated the target alarm data, where the target alarm data refers to the alarm data for the target AI visual analysis service. If the AI ​​middle station has generated at least one target alarm data, the fault location can be determined based on the subscription information of the target AI visual analysis service and the push log of the target AI visual analysis service to determine the fault location result.

[0059] If the AI ​​middle station does not generate the target alarm data, you can view the task details of the AI ​​middle station, the status of the equipment used to run the target AI visual analysis business (i.e., the target device status), and the task status on the algorithm side (i.e., the target algorithm status) respectively. Among them, the task details of the AI ​​middle station can be used to reflect the task status on the platform side, and the target algorithm status can be used to reflect the task status on the algorithm side. In the case that the AI ​​middle station does not generate the target alarm data, this application can locate the fault based on the task details, target device status, and target algorithm status, and determine the fault location result.

[0060] After determining the fault location result, the present application can generate a fault troubleshooting result based on the fault location result and push the fault troubleshooting result to relevant personnel. In some embodiments, the fault troubleshooting result may include the fault location result, fault cause analysis and repair guidance suggestions.

[0061] Furthermore, under certain conditions, the present application can start the automatic fault repair function, for example, by restarting the service, switching to backup equipment, horizontal expansion and contraction, etc., to achieve an automated processing mechanism, thereby further reducing manual intervention. In some examples, the present application can use one of the following two methods to achieve the automatic fault repair function:

[0062] (1) Push the troubleshooting results to the external system through Webhook, and the external system can perform corresponding actions based on the content of the troubleshooting results;

[0063] (2) Distribute the repair script to the alarm device (i.e., the faulty device) or the designated detection target point, so that the alarm device executes the repair script to achieve fault repair. The repair script can be a shell script, a python script, a bat script, etc.

[0064] If you want to automatically trigger the execution of the repair script on the alarm device, you need to install a probe at the detection target point. This probe periodically initiates heartbeats to the server, pulls the script to be executed, and reports the execution result of the script to the server. If the automatic execution of the fault repair action is enabled, when an alarm event occurs, not only will the details of the script used for fault repair be displayed, but the server will also create an execution task and wait for the probe to pull and execute it. When the fault repair notification is enabled, when the fault recovery strategy is met, the restored monitoring items, the stdout of the automatic triggering of the execution of the repair script, and the measurement values ​​of the monitoring data will be transmitted to the alarm terminal to inform that the fault has been repaired.

[0065] Usually, one alarm corresponds to one repair script, or the repair script can be made more general to cope with different scenarios. In some examples, the key information of the alarm event (such as detection target point, alarm strategy name, etc.) can be transmitted to the repair script through the standard input stdin. After receiving the content of the alarm event, the repair script can make different processing.

[0066] Furthermore, in the case of a business about to fail or a potential failure, this application can proactively discover and warn in the first place, and can also automatically conduct in-depth analysis, and accurately describe the method of eliminating the potential failure through the early warning notification method. For proactive discovery of the fault that is occurring, it can not only automatically perform preliminary positioning, accurately narrow down and describe the scope of the fault point through the alarm notification method, but also automatically conduct in-depth analysis, and accurately and in detail describe the cause of the fault and the method of repairing the fault through the alarm method. While transmitting the data to be transmitted to the alarm terminal in an alarm manner, the fault can be automatically repaired if conditions permit. When an alarm event occurs, not only will the details of the script used for fault repair be displayed, but the server will also create an execution task, waiting for the probe to pull and execute. When the fault repair notification is enabled, when the fault recovery strategy is met, the restored monitoring items, the standard output stdout of the automatic triggering execution of the repair script, and the metric values ​​of the monitoring data will be transmitted to the alarm terminal to inform that the fault has been repaired. It can intuitively and efficiently perform closed-loop tracking of troubleshooting of these alarm problems.

[0067] In some embodiments, Figure 2 As shown in the figure, the fault is located according to the subscription information and push logs of the target AI visual analysis service, and the fault location results are obtained, including:

[0068] Determine whether the target AI visual analysis service is subscribed and whether there are any push anomalies based on the subscription information and push logs;

[0069] If the target AI visual analysis service is not subscribed or has push exceptions, the application problem will be used as the fault location result;

[0070] If the target AI visual analysis service is subscribed and the push log is empty, the AI ​​middle station push service exception is used as the fault location result.

[0071] Among them, the subscription information of the target AI visual analysis service may record the subscriber information of the target AI visual analysis service and / or be used to reflect whether the target AI visual analysis service is subscribed. The push log of the target AI visual analysis service may be used to record the situation in which the target AI visual analysis service pushes messages to subscribers.

[0072] Specifically, when the AI ​​middle station has target alarm data, the subscription information and push log of the target AI visual analysis service can be viewed separately. If the target AI visual analysis service is not subscribed, or there is a push abnormality problem with the target AI visual analysis service, it can be determined that a fault occurred on the application side, so the application problem can be used as the fault location result. In one example, after determining that it is an application problem, the application can feedback the relevant fault to the application.

[0073] If the target AI visual analysis service has a subscription but no corresponding message push, it means that the application side has received the subscription but cannot complete the message push. Therefore, it can be determined that there is an abnormality in the push service of the AI ​​middle station. Therefore, the abnormality of the AI ​​middle station push service can be used as the fault location result. Furthermore, after determining that it is an issue with the AI ​​middle station push service, this application can troubleshoot the AI ​​middle station microservices for a more in-depth analysis, and then accurately and in detail find out the cause of the failure.

[0074] In this way, the accuracy of the fault location result can be further improved, thereby further improving the accuracy of the fault handling.

[0075] In some embodiments, the task details include a target task status for a target AI visual analysis service generated by the AI ​​platform, ie, the task status on the platform side.

[0076] like Figure 2 As shown, the fault location is performed according to the task details of the AI ​​middle station, the target device status, and the target algorithm status corresponding to the target AI visual analysis service, and the fault location results are obtained, including:

[0077] If the target task status is normal and the target algorithm status is normal, then if the service type of the target AI visual analysis service is a video service, the algorithm-side problem is used as the fault location result;

[0078] If the target task status is normal and the target algorithm status is normal, then if the service type is a frame extraction service and there is a call log, the algorithm side problem is used as the fault location result; and if the service type is a frame extraction service and there is no call log, the algorithm stream extraction problem is used as the fault location result;

[0079] If the target task status is abnormal, and the target algorithm status is push abnormality / the target device status is offline, then try to perform a capture operation on the video stream involved in the target AI visual analysis business and obtain the video stream address;

[0080] If the capture is successful, the algorithm's stream pulling and frame extraction problem is used as the fault location result;

[0081] If the capture fails and the video stream address acquisition fails, the video aggregation platform problem / device problem will be used as the fault location result;

[0082] If the capture fails and the video stream address is obtained successfully, the failure log is used to determine whether the frame extraction algorithm can provide normal service. If the frame extraction algorithm can provide normal service, the network problem / stream problem is used as the fault location result. If the frame extraction algorithm service is abnormal, the AI ​​middle station problem is used as the fault location result.

[0083] Specifically, when the AI ​​middle station has no target alarm data, this application can view the task details of the AI ​​middle station to determine the task status of the target AI analysis service on the platform side. This application can also view the target algorithm type corresponding to the target AI visual analysis service to determine the task status on the algorithm side. In addition, this application also checks the device status.

[0084] When the target task status is normal and the target algorithm status is normal, it indicates that both the platform side and the algorithm side believe that the target AI visual analysis service is executing normally. In this case, analysis can be performed based on the business type of the target AI visual analysis service. If the business type of the target AI visual analysis service is a video service, it can be determined that the problem is on the algorithm side, and the problem on the algorithm side can be used as the fault location result. Furthermore, the present application can troubleshoot the algorithm or network connection to deeply analyze the cause of the fault.

[0085] When the target task status is normal and the target algorithm status is normal, if the service type of the target AI visual analysis service is a frame extraction service, you can further view the call log of the target AI visual analysis service. If the target AI visual analysis service has a call log, it can be determined that the problem is on the algorithm side. Furthermore, the algorithm or network connection can be checked to further analyze the cause of the failure. If there is no call log, it means that the target AI visual analysis service has failed to successfully pull the stream and extract the frame. Therefore, it can be determined that the problem is with the algorithm pulling the stream and extracting the frame. Furthermore, the frame extraction algorithm or network can be checked to further analyze the cause of the failure.

[0086] If the target task status is abnormal and the target algorithm status is push abnormal, try to perform a snapshot operation on the video stream involved in the target AI visual analysis service and obtain the video stream address. If the target task status is abnormal and the target device status is offline, try to perform a snapshot operation on the video stream involved in the target AI visual analysis service and obtain the video stream address.

[0087] If the video stream involved in the target AI visual analysis business is captured successfully, it can be determined that the algorithm is pulling the stream and extracting frames. Therefore, the algorithm pulling the stream and extracting frames can be used as the fault location result. If the capture is unsuccessful, further analysis can be performed based on the acquisition result of the video stream address. If the video stream address acquisition fails, it can be determined that the problem is the video aggregation platform / equipment problem. Therefore, the video aggregation platform / equipment problem can be used as the fault location result. Furthermore, feedback can be provided to the video aggregation platform so that the video aggregation platform can conduct inspection and in-depth analysis.

[0088] If the capture fails and the video stream address is obtained successfully, the frame extraction algorithm is judged according to the failure log whether it can serve normally. If the frame extraction algorithm serves normally, it can be determined that it is a network problem or a stream problem. If the frame extraction algorithm fails, it can be determined that it is a problem on the AI ​​middle station side. Furthermore, if the frame extraction algorithm fails, this application can operate the frame extraction algorithm for recovery dialing purposes.

[0089] Furthermore, if the target task status is normal and the target algorithm status is that the task does not exist, the present application can operate the recovery algorithm task. In the case where the recovery algorithm task returns a failure, it can be determined that it is a problem on the AI ​​middle station side.

[0090] In this way, the accuracy of the fault location result can be further improved, thereby further improving the accuracy of the fault handling.

[0091] In some embodiments, the automated monitoring and operation method of an AI visual analysis service provided by the present application further includes the following steps:

[0092] According to the preset multiple monitoring indicators and the preset collection cycle, the indicator data of the AI ​​visual analysis service is collected; wherein the multiple monitoring indicators include service quality indicators and business indicators;

[0093] According to the preset inspection cycle, automatic inspection is carried out according to the indicator data, and the automatic inspection results are obtained; the automatic inspection results are normal or abnormal.

[0094] Specifically, this application can combine the indicator class of monitoring service quality goals with the indicator class of AI visual analysis business, attach importance to the construction and monitoring of the business indicator system, enumerate the result indicators and fault categories corresponding to the core business functions, and thus obtain multiple monitoring indicators. In an example, the various monitoring indicators of this application can be as follows Figure 3A and Figure 3B As shown, Figure 3A and Figure 3B Business-related indicators and service quality-related indicators are shown.

[0095] Similar to the commonly used IT architecture, the server side of AI visual analysis business generally includes the following four layers:

[0096] (1) Functional interfaces: such as login interfaces, subscription interfaces, alarm push interfaces, etc., usually hundreds or thousands;

[0097] (2) Microservices: such as user management programs, AI vision manufacturer management programs, summary display of AI vision analysis results, programs for pushing alarm data, and other program services developed, written and run;

[0098] (3) Standard middleware: such as MySQL, Redis, Kafka, etc.

[0099] (4) Infrastructure: such as switches, cloud hosts, video equipment, etc.

[0100] All objects are layered and grouped, and the health of each type of object is quantified. Users are guided to analyze the indicators, logs, links, and events of abnormal objects and combine them into visual monitoring to facilitate the analysis and confirmation of faults.

[0101] When multiple monitoring indicators are set, the present application can collect indicator data of the AI ​​visual analysis service according to a preset collection cycle. The collected indicator data includes indicator data corresponding to service quality indicators and indicator data corresponding to business indicators.

[0102] Based on the collected indicator data, this application can perform automated inspections on the target AI visual analysis service according to pre-set inspection cycles and preset alarm rules to determine whether there are any alarms for the target AI visual analysis service and obtain automated inspection results.

[0103] In an example, the following briefly describes the designation of alarm rules, taking the feedback from a tenant who has subscribed to the AI ​​visual analysis service that he has not received the detection and analysis results of the subscribed AI visual atomic capabilities as an example. Figure 4 and Figure 5 This application can automatically inspect all tasks of the AI ​​vision business service at regular intervals. If the query detects one of the following situations, the alarm status of the task is determined to be normal, and the metric value is marked as "0". There is no need to start the fault troubleshooting process and alarm notification:

[0104] (1) The task has no corresponding subscription information, and the alarm data on the platform side is successfully pushed to the view library;

[0105] (2) The task has corresponding subscription information, and the subscription address contains a string that contains the internal whitelist (this means that the application side obtains the alarm data by subscribing to the view library), and the alarm data on the platform side is successfully pushed to the view library;

[0106] (3) The task has corresponding subscription information, and the subscription address does not contain a string in the internal whitelist (this means that the application side obtains the alarm data through the subscription platform), and the alarm data on the platform side is successfully pushed to the address;

[0107] (4) For video tasks, the platform does not find the alarm data for the task on that day, but the task status on the platform side is normal, and the task in the dial query algorithm container exists and is in normal status. According to the location of the troubleshooting layer, the range convergence may be a problem with the image algorithm. In this case, the current time point can be determined. If the current time point does not exceed the preset time (for example, 12 noon), it means that it is still less than a complete time period (for example, one day) from the last inspection (or the time when the task was put into use), so no alarm can be issued for the time being.

[0108] If the current time point exceeds the preset time, the total number of alarms for the task in the previous complete time period (hereinafter referred to as the first total number) can be queried and compared with the total number of alarms in the current complete time (hereinafter referred to as the second total number). When the difference between the first total number and the second total number is greater than the preset threshold, it is determined that the task is abnormal and an alarm is issued, otherwise, no alarm is temporarily issued.

[0109] (5) For tasks such as frame extraction and arrangement, the platform does not find the alarm data for the task on that day, but the task status on the platform side is normal, the task in the dial-up query algorithm service exists and is in normal status, and there are also logs for calling image algorithms. According to the location of the troubleshooting process, the range convergence may be a problem with the image algorithm. In this case, the current time point can be determined. If the current time point does not exceed the preset time (for example, 12 noon), it means that it is still less than a complete time period (for example, one day) from the last inspection (or the time when the task was put into use), so no alarm can be issued for the time being.

[0110] If the current time point exceeds the preset time, the total number of alarms for the task in the previous complete time period (hereinafter referred to as the third total number) can be queried and compared with the total number of alarms in the current complete time (hereinafter referred to as the fourth total number). When the difference between the third total number and the fourth total number is greater than the preset threshold, it is determined that the task is abnormal and an alarm is issued, otherwise, no alarm is temporarily issued.

[0111] If the tasks do not meet the above conditions, the metric value is marked as "1", and the troubleshooting process needs to be started to automatically conduct in-depth analysis to accurately and in detail find out the cause of the fault, and transmit the in-depth analysis of the fault cause and the method of repairing the fault to the alarm terminal in the form of an alarm. For the description of the troubleshooting process, please refer to the above embodiment, which will not be repeated here.

[0112] In some embodiments, collecting indicator data of AI visual analysis services includes:

[0113] Multiple probe collectors are used to collect indicator data of AI visual analysis services respectively; among them, multiple probe collectors reuse the scrape_configs configuration item of the Promethus tool for configuration.

[0114] In this embodiment, considering the diversity of monitoring indicators, this application needs to use multiple probe collectors to collect indicator data. However, in the field of monitoring data collection, there are many probe collectors in the original ecology. Although the ecology is prosperous, it cannot be used out of the box, and the log library, configuration file management method, and command line parameter transmission method used by the probe collector are different, and the experience is different. Therefore, this application integrates all detection and collection tools related to AI visual business analysis and encapsulates them into a unified monitoring probe collector. The collector can fully cover all monitoring targets, realize centralized management, simplify the configuration process, and allow users to directly specify the capture target details, capture rules, etc. in the configuration of the monitoring probe collector, thereby automatically completing the collection, cleaning and classification of monitoring data.

[0115] In addition, by formulating standardized templates for configuration files, not only are the problems of different configuration file formats and uneven operating experiences between different plug-ins eliminated, but the learning threshold for the technical team is also significantly lowered, which promotes the rapid deployment and delivery of projects and better adapts to the rapid changes in market and business needs. The unified monitoring probe collector can not only integrate third-party plug-ins, but also expand custom monitoring targets, obtain data from the database form of the business, and generate key business indicators based on preset algorithms, such as the use of business core function service calls and AI visual alarm push volume, etc. According to the pre-established data specifications, they are converted into a unified format, and these data are connected in series and aggregated into business indicators. Based on the needs of the business, the indicators are intelligently detected for anomalies, abnormal trends in business indicators are found, and alarms are issued in a timely manner, so as to truly turn passivity into initiative and discover fault problems before complaints.

[0116] Specifically, this application can modify the monitoring target configuration items that the unified monitoring probe collector needs to collect according to the visual troubleshooting flowchart, specify the information of the capture target, the capture rules, etc., build the specifications of the monitoring data, and do a good job of cleaning, classifying and persistent storage of the monitoring data. This configuration reuses Promethus's scrape_configs as much as possible. Each scrape_configs configuration contains a job name, target address, capture interval, timeout and other parameters, thereby reducing the learning cost. In addition, scrape_configs can also be expanded. For example, when monitoring the indicators of a table in a specified database, some SQL statements must be configured, and these are the collection rules. These collection rules are referenced in scrape_configs, and these collection rules are managed in a file-cutting manner. In this way, different collection rules can be used for different detection targets, which is very flexible.

[0117] Furthermore, all configurations are placed in a directory file. Each subdirectory under this file represents a type of plug-in. For example, a kafka directory under the configuration directory contains configurations related to the kafka collection plug-in, and an elasticsearch directory contains configurations related to the elasticsearch collection plug-in. For a plug-in directory, there will be one or more entry configuration files. When the monitoring probe collector matches the entry configuration file, it will read and assemble the collection rules corresponding to the collection targets configured in these entry configuration files in sequence and then capture the data after forming a configuration file. After the unified monitoring probe collector collects the data, it will eventually push it to the remote time series library through the prometheus remotewrite protocol. Multiple push addresses can be configured. If a backend only wants to receive specific data, it can reset the label configuration through relabel for filtering. Different push addresses can be attached with different label labels, and different relabel rules can also be configured.

[0118] The main purpose of the unified probe collector is to manage all monitoring indicators and unify the different writing styles of configuration items at different monitoring points, so that it can be used out of the box. Some core indicators are obtained from the detection points and encapsulated as detection metrics of the time series database, and then the unified monitoring probe collector writes them to the remote time series database for persistent storage of the data.

[0119] In addition, this application can not only integrate third-party plug-ins, but also expand custom monitoring indicators. The specific examples listed above are to obtain data from the database form of the business and calculate and generate business indicators. Custom monitoring indicators can also be the usage of business core function interface calls and the push volume of AI visual analysis results, etc. According to the pre-established data specifications, such as determining the target of log records, log levels, applications and services use a consistent and structured log format, and unstructured logs are converted into a unified format. These data are connected in series, and the contextual information associated with the logs is recorded in detail and summarized into business indicators. In combination with business needs, the indicators are intelligently detected for anomalies, abnormal business indicator trends are found, and alarms are issued in time to truly transform passivity into initiative and discover fault problems before complaints.

[0120] At present, the existing technologies generally have a poor alarm experience. The main manifestation is that the operation and maintenance team may need to deal with hundreds or even more alarms every day. In many cases, only a few of them are effective and need to be handled immediately. These key alarms are often overwhelmed by a large number of false alarms and are difficult to identify in time. What's more difficult is that different roles such as operations, R&D, and customers have different concerns about system failures, which makes it difficult for each position to respond to and handle real problems in the first time.

[0121] In this application, in order to avoid false alarms and repeated alarms, alarm noise reduction rules are set, such as sending only one notification when the same alarm is triggered multiple times in a short period of time. The historical alarm data is analyzed by algorithms to identify common false alarm patterns and causes, and the alarm noise reduction rules are adjusted accordingly. In the actual operation process, the alarm noise reduction rules are continuously tested and optimized to ensure that they can effectively reduce the false alarm rate and repeated alarm rate.

[0122] In addition, setting reasonable alarm noise reduction and alarm suppression strategies to aggregate similar alarms for dispatch, notification, and processing not only improves efficiency but also reduces alarm fatigue and the proliferation of false alarms.

[0123] Specifically, in some embodiments, automatic inspection is performed according to the indicator data, and automatic inspection results are obtained, including:

[0124] Carry out automated inspections based on indicator data;

[0125] If at least one target alarm data is generated during the automated inspection process, historical alarm data of the target AI visual analysis service is obtained, and at least one target alarm data is filtered based on the historical alarm data to obtain a filtering result;

[0126] If the filter result is empty, the automatic inspection result is normal, otherwise, the automatic inspection result is abnormal.

[0127] In this embodiment, during the automated inspection process, current alarm data can be filtered and screened based on historical alarm data, thereby achieving alarm data noise reduction. In this way, alarm noise reduction rules can be automatically determined, thereby improving the degree of automation.

[0128] Specifically, the present application can use historical data to predict the current state of the target AI visual analysis service, and use historical alarm data to predict whether the current new alarm needs to be filtered out. This method can involve a variety of algorithm models, including but not limited to any one or any combination of algorithm models such as similarity models, classification models, and prediction models.

[0129] The process of intelligent noise reduction does not require too much configuration by the user. It only requires setting the alarm classification and priority, configuring the frequency and duration of the execution of the monitoring detection point target indicator data combined with the year-on-year ring, aggregation and other operator statistical formulas. Compared with the noise reduction mechanism that only sets the threshold / dynamic threshold and sets the default rule, this noise reduction mechanism provides higher flexibility and automation. In some examples, the specific implementation steps can be as follows:

[0130] Step A1: Alarm classification and priority setting: Classify alarms by type and set different priorities according to their severity and urgency to help operation and maintenance personnel better understand and handle alarm information, and give priority to those alarms that have the greatest impact on system stability and business.

[0131] Step A2: Setting of abnormal threshold, execution frequency and execution duration: By combining operator formulas such as year-on-year cycle and aggregation, reasonable abnormal threshold, execution frequency and execution duration of statistical data characteristics are set to avoid interference of short-term fluctuations on alarm services. Alarms are triggered only when indicators exceed a certain threshold for a period of time, thereby reducing false alarms.

[0132] Step A3: Alarm deduplication and aggregation: Deduplication or aggregation is used for the same or similar alarms, and only one related alarm message is retained or merged into one alarm message, so as to reduce the interference of duplicate alarms to operation and maintenance personnel, and also reduce the storage and processing pressure of alarm services.

[0133] Step A4: Alarm shielding and shielding rules: For known irrelevant alarms or alarms that do not need to be processed temporarily, shielding rules are set to shield them to avoid interference to operation and maintenance personnel. Shielding rules can be set based on alarm type, source, time and other conditions.

[0134] In some embodiments, automatic inspection is performed based on the indicator data, and automatic inspection results are obtained, including:

[0135] Carry out automated inspections based on indicator data;

[0136] If at least one target alarm data is generated during the automated inspection process, the at least one target alarm data is subjected to noise reduction filtering according to the alarm noise reduction rule configured in advance through the low-code platform to obtain a filtering result;

[0137] If the filter result is empty, the automatic inspection result is normal, otherwise, the automatic inspection result is abnormal.

[0138] In this embodiment, the low-code platform can be used to quickly build and change the notification method of the alarm, reasonably configure alarm noise reduction, alarm suppression strategy and visual monitoring template, simplify the configuration of monitoring rules, lower the technical threshold and reduce maintenance costs. At the same time, users can also subscribe to the monitoring rules related to themselves according to different user roles (operation and maintenance, operation or R&D personnel), subscribe to unified and standardized monitoring rules, and redefine the alarm level, notification method and callback interface, thereby improving the alarm response speed. Better adapt to the ever-changing market environment and accelerate the promotion of AI visual analysis business.

[0139] In some embodiments, generating and pushing a fault troubleshooting result based on the fault location result includes:

[0140] If the filtering result is not empty, the target alarm data in the filtering result is pushed to the alarm push user according to the alarm push rule customized by the alarm push user; wherein the alarm push rule includes at least one of the following: alarm level, notification method and callback interface.

[0141] In this application, when a business is about to fail or has a potential failure, it will be monitored and proactively discovered and warned in the first place, and can also automatically conduct in-depth analysis, accurately describing how to eliminate the potential failure through early warning notifications. For proactively discovered failures that are occurring, it can automatically perform preliminary positioning, and configure a variety of notification methods according to actual needs, such as text messages, emails, phone calls, instant messaging software, etc. Ensure the reliability and timeliness of the notification method, and be able to accurately narrow down and describe the scope of the failure point and convey the warning information to relevant personnel in the first place.

[0142] In addition, the present application can also automatically conduct in-depth analysis, and accurately and in detail describe the cause of the failure, the method of repairing the failure, and the executable script that automatically completes the fault repair by means of an alarm. While transmitting the data to be transmitted to the alarm terminal in the form of an alarm, the function of automatically completing the repair of the fault can be started if conditions permit, reducing manual intervention. In addition, the historical alarm data is analyzed through an algorithm to identify common false alarm patterns and causes, and the alarm noise reduction rules are adjusted in a low-code manner accordingly. The alarm noise reduction and alarm suppression strategies are reasonably configured and modified, and similar alarms are aggregated together for dispatch, notification, and processing, which greatly improves efficiency, reduces alarm fatigue, and can intuitively and efficiently perform closed-loop tracking of troubleshooting of these alarm problems.

[0143] For different user roles (such as maintenance personnel, operations personnel or R&D personnel), this application can subscribe to monitoring rules related to the user according to different user roles, subscribe to unified and standardized monitoring rule templates, redefine the alarm level, notification method and callback interface, and improve the alarm response speed. In some examples, the specific implementation steps can be as follows:

[0144] Step B1: Set the subscription name and configure the imported alarm policy template to be subscribed;

[0145] Step B2: The alarm monitoring metrics can be re-filtered and screened. If the match is successful, the data is transmitted to the alarm terminal. If the match fails, the data is not transmitted to the alarm terminal.

[0146] Step B3: You can set the duration of the subscription event. For example, if the duration is configured to be 100 seconds, the same alarm event will not be considered a successfully matched subscription event when it is subscribed for the first time. When the subscription event is matched again, the difference between the trigger time of the current subscription event and the trigger time of the first subscription of this event will be calculated. If the value obtained exceeds 100 seconds, it will be considered that the subscription condition has been met and the alarm notification logic process will be followed. If it is less than 100 seconds, the subscription event will not be considered to have been successfully matched and the alarm notification logic process will not be followed. This function can be used as an alarm upgrade. The team leader can configure a subscription that lasts more than 1 hour (3600s) and the recipient is configured as the project leader. As the project leader, it is ensured that someone will follow up on the alarm. If the fault cannot be solved for more than 1 hour, the fault level needs to be upgraded and reported to your leader.

[0147] Step B4: Based on the requirements, the alarm level, alarm notification method and callback interface address can be redefined.

[0148] In some embodiments, the indicator data includes the indicator value, the device identifier of the indicator source device, and the algorithm code of the target AI visual analysis service. The automated monitoring and operation method of the AI ​​visual analysis service provided in this application may also include the following steps:

[0149] Build a visual dependency graph based on the device identification and algorithm coding of each indicator data;

[0150] Generate a visual fault diagnosis path based on the visual dependency graph and each indicator value.

[0151] In this embodiment, in the process of collecting indicator data, a correspondence between the indicator value, device identification and algorithm code can be generated to facilitate data concatenation based on the correspondence and construct a visual dependency graph, thereby solving the problem of data islands.

[0152] In one example, when a user creates a task, the task may be assigned a unique task identifier, and associated with the device information (i.e., device identifier) ​​and the type of AI visual atomic capability (i.e., algorithm code) that the value depends on. This application may combine the task identifier, device identifier, and algorithm code into indicator data, so that a set of indicator data becomes an "observation object", and determine the health status of the "observation object" in combination with the policy logic shown above.

[0153] In some embodiments, see Figure 5 ,This application can split the functions into multiple sub-modules and implement automated inspection accordingly.

[0154] The persistent time series database brings together disorganized data, massive indicators, logs, and link tracking data. At present, the common monitoring and observation platforms in the industry basically present the data in a flat format, and only simply classify them into indicators, logs, and link tracking through the function menu, and then further subdivide them into sub-items such as infrastructure, applications, and business under the indicator category. This application pays more attention to the intrinsic connection and dynamic association between data. Users do not need to build complex global models in their minds, and can easily achieve seamless connection and deep insight of data through the intelligent linkage function. By cleverly integrating monitoring service quality target indicators and AI visual business indicators, a comprehensive and detailed AI visual business indicator system has been constructed. Enumerate the result indicators and fault categories corresponding to the core business functions, and make good monitoring rules for the result indicators to ensure that the coverage system will be alarmed if there is a fault.

[0155] This application uses low-code platform tools and data operations to present data features (maximum value, minimum value, year-on-year and month-on-month changes, etc.), establish preliminary monitoring indicators (whether the network is normal, whether the middleware service is normal, whether there are any problems with the dependent third-party services, etc.), and then integrate all monitoring indicators to build a visual dependency graph to form a better knowledge accumulation, which can be observed and monitored at any time, and finally draw a conclusion on fault location. It can clearly know which module of the AI ​​visual analysis service caused the current failure. Locate the direct cause of the failure, find the basis for stopping the loss, and stop the loss as soon as possible.

[0156] The method adopted in this application is a reuse strategy based on knowledge accumulation, which aims to guide users to first focus on key data for analysis when locating faults, then examine related data in a step-by-step manner, and finally integrate and present various types of data to form a comprehensive and clear fault diagnosis path.

[0157] First, provide a global cockpit perspective so that users can see the health status of each business at a glance.

[0158] If the health status of a business is red, it means there is an abnormality. Click Details to further view which core business indicator is abnormal.

[0159] If the core business indicators plummet, and at certain moments the indicators directly show "0", this is an obvious fault. Users can click the mouse at the location of the plummet to see whether the associated services (functional interfaces, microservices, middleware or infrastructure) are healthy. If they are unhealthy, they will be marked in red, and users can quickly locate the faulty service.

[0160] The user clicks the red mark to continue drilling down for troubleshooting. They can view logs, link data, etc., view related logs based on abnormal indicators, and view link data based on the traceID in the log. Then, click the trace button to connect the log to the link data and finally find the root cause of the fault.

[0161] During the whole process, users only need to click a few times, and can quickly locate the fault according to the color guidance of the health status. Even maintenance, operation, and R&D personnel with zero basic knowledge or who are not familiar with the entire system can quickly find out the root cause of the fault through the above guidance method.

[0162] The following describes the automated monitoring and operation and maintenance device for the AI ​​visual analysis service provided in an embodiment of the present application. The automated monitoring and operation and maintenance device for the AI ​​visual analysis service described below and the automated monitoring and operation and maintenance method for the AI ​​visual analysis service described above can be referenced to each other.

[0163] In one embodiment, Figure 6 As shown, the automatic monitoring and operation device 700 of the AI ​​visual analysis service of the present application includes:

[0164] The judgment module 702 is used to judge whether the AI ​​middle station generates target alarm data for the target AI visual analysis service in response to the fault troubleshooting interface being called or the automatic inspection result being abnormal; wherein the target AI visual analysis service is the AI ​​visual analysis service that needs to be troubleshooted;

[0165] The first fault location module 704 is used to locate the fault according to the subscription information and push log of the target AI visual analysis service if the AI ​​middle station generates the target alarm data, and obtain the fault location result;

[0166] The second fault location module 706 is used to locate the fault according to the task details, target device status and target algorithm status corresponding to the target AI visual analysis service of the AI ​​middle station if the target alarm data is not generated by the AI ​​middle station, and obtain the fault location result; wherein the target device status is the status of the device used to run the target AI visual analysis service;

[0167] The push module 708 is used to generate and push a fault troubleshooting result according to the fault location result.

[0168] In one embodiment, the first fault location module of the present application includes:

[0169] A subscription push judgment unit, used to judge whether the target AI visual analysis service is subscribed and whether there is a push anomaly according to the subscription information and the push log;

[0170] A first positioning result acquisition unit is used to use the application problem as the fault positioning result if the target AI visual analysis service is not subscribed or there is a push exception;

[0171] The second positioning result acquisition unit is used to take the AI ​​middle station push service exception as the fault positioning result if the target AI visual analysis service is subscribed and the push log is empty.

[0172] In one embodiment, the task details include a target task status for the target AI visual analysis service generated by the AI ​​platform. The second fault location module of the present application includes:

[0173] A third positioning result acquisition unit is used for, if the target task state is normal and the target algorithm state is normal, and when the service type of the target AI visual analysis service is a video service, taking the algorithm side problem as the fault positioning result;

[0174] A fourth positioning result acquisition unit is used for, if the target task state is normal and the target algorithm state is normal, then, when the service type is a frame extraction service and there is a call log, taking the algorithm side problem as the fault location result, and when the service type is a frame extraction service and there is no call log, taking the algorithm stream extraction problem as the fault location result;

[0175] A capture unit, configured to attempt to perform a capture operation on the video stream involved in the target AI visual analysis service and obtain the video stream address if the target task status is abnormal and the target algorithm status is push abnormality / the target device status is offline;

[0176] A fifth positioning result acquisition unit, configured to take the problem of algorithm stream pulling and frame extraction as the fault positioning result if the capture is successful;

[0177] A sixth positioning result acquisition unit, configured to take a video aggregation platform problem / equipment problem as the fault positioning result if the capture fails and the video stream address acquisition fails;

[0178] The seventh positioning result acquisition unit is used to determine whether the frame extraction algorithm can provide normal service based on the failure log if the capture fails and the video stream address is successfully obtained. If the frame extraction algorithm can provide normal service, the network problem / stream problem will be used as the fault location result; and if the frame extraction algorithm service is abnormal, the AI ​​middle station problem will be used as the fault location result.

[0179] In some embodiments, the automated monitoring and operation device for the AI ​​visual analysis service further includes:

[0180] An indicator collection module, used to collect indicator data of the AI ​​visual analysis service according to a preset plurality of monitoring indicators and a preset collection cycle; wherein the plurality of monitoring indicators include service quality indicators and business indicators;

[0181] The automatic inspection module is used to perform automatic inspection according to the indicator data in accordance with a preset inspection cycle and obtain the automatic inspection result; the automatic inspection result is normal or abnormal.

[0182] In one embodiment, the indicator collection module of the present application includes:

[0183] A multi-probe collection unit is used to use multiple probe collectors to respectively collect indicator data of the AI ​​visual analysis business; wherein the multiple probe collectors reuse the scrape_configs configuration item of the Promethus tool for configuration.

[0184] In some embodiments, the automatic inspection module of the present application includes:

[0185] A first inspection unit, used for performing automated inspection according to the indicator data;

[0186] A first filtering unit is used to obtain historical alarm data of the target AI visual analysis service if at least one target alarm data is generated during the automated inspection process, and filter the at least one target alarm data based on the historical alarm data to obtain a filtering result;

[0187] The first inspection result obtaining unit is used to determine that if the filtering result is empty, the automatic inspection result is normal, otherwise, the automatic inspection result is abnormal.

[0188] In one embodiment, the automatic inspection module of the present application includes:

[0189] A second inspection unit, used for performing automated inspection according to the indicator data;

[0190] A second filtering unit is configured to, if at least one target alarm data is generated during the automated inspection process, perform noise reduction filtering on the at least one target alarm data according to an alarm noise reduction rule pre-configured through the low-code platform to obtain a filtering result;

[0191] The second inspection result obtaining unit is used to determine that if the filtering result is empty, the automatic inspection result is normal, otherwise, the automatic inspection result is abnormal.

[0192] In some embodiments, the push module of the present application includes:

[0193] The alarm push unit is used to push the target alarm data in the filtering result to the alarm push user according to the alarm push rules customized by the alarm push user if the filtering result is not empty; wherein the alarm push rules include at least one of the following: alarm level, notification method and callback interface.

[0194] In one embodiment, the indicator data includes the indicator value, the device identifier of the indicator source device and the algorithm code of the target AI visual analysis service. The automatic monitoring and operation and maintenance device of the AI ​​visual analysis service of the present application also includes:

[0195] A dependency graph construction module, used to construct a visual dependency graph according to the device identification and algorithm coding of each of the indicator data;

[0196] The diagnostic path generation module is used to generate a visual fault diagnostic path based on the visual dependency graph and each of the indicator values.

[0197] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the automated monitoring and operation method of the AI ​​visual analysis service in any embodiment.

[0198] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the automated monitoring and operation method for AI visual analysis services in any embodiment.

[0199] Indicatively, Figure 7 The internal structure diagram of a computer device provided in an embodiment of the present application is shown in FIG. 1 . In one example, the computer device may be a server. Figure 7 , the computer device 900 includes a processing component 902, which further includes one or more processors, and a memory resource represented by a memory 901, for storing instructions executable by the processing component 902, such as an application. The application stored in the memory 901 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 902 is configured to execute instructions to perform the steps of the automated monitoring and operation method of the AI ​​visual analysis service described in any of the above embodiments.

[0200] The computer device 900 may further include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate based on an operating system stored in the memory 901, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0201] Those skilled in the art will understand that the internal structure of the computer device shown in the present application is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0202] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.

[0203] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0204] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automated monitoring and operation method for AI visual analysis services, characterized in that: include: In response to the troubleshooting interface being called or the automatic inspection result being abnormal, determining whether the AI ​​middle station generates target alarm data for the target AI visual analysis service; wherein the target AI visual analysis service is the AI ​​visual analysis service that needs troubleshooting; If the AI ​​middle station generates the target alarm data, the fault location is performed according to the subscription information and push log of the target AI visual analysis service, and the fault location result is obtained; If the AI ​​middle station does not generate the target alarm data, the fault location is performed according to the task details of the AI ​​middle station, the target device status and the target algorithm status corresponding to the target AI visual analysis service, and the fault location result is obtained; wherein the target device status is the status of the device used to run the target AI visual analysis service; Generate and push a fault troubleshooting result based on the fault location result.

2. The method according to claim 1, characterized in that The fault location is performed according to the subscription information and push log of the target AI visual analysis service, and the fault location result is obtained, including: Determine whether the target AI visual analysis service is subscribed and whether there is a push anomaly according to the subscription information and the push log; If the target AI visual analysis service is not subscribed or has a push exception, the application problem is used as the fault location result; If the target AI visual analysis service is subscribed and the push log is empty, the AI ​​middle station push service exception is used as the fault location result.

3. The method according to claim 1, characterized in that The task details include a target task status generated by the AI ​​platform for the target AI visual analysis service; The fault location is performed according to the task details of the AI ​​middle station, the target device status, and the target algorithm status corresponding to the target AI visual analysis service, and the fault location result is obtained, including: If the target task status is normal and the target algorithm status is normal, then when the service type of the target AI visual analysis service is a video service, the algorithm-side problem is used as the fault location result; If the target task status is normal and the target algorithm status is normal, then when the service type is a frame extraction service and there is a call log, the algorithm side problem is used as the fault location result, and when the service type is a frame extraction service and there is no call log, the algorithm stream extraction problem is used as the fault location result; If the target task status is abnormal, and the target algorithm status is push abnormality / the target device status is offline, then try to perform a capture operation on the video stream involved in the target AI visual analysis service and obtain the video stream address; If the capture is successful, the algorithm stream pulling frame extraction problem is used as the fault location result; If the capture fails and the video stream address acquisition fails, the video aggregation platform problem / device problem is taken as the fault location result; If the capture fails and the video stream address is obtained successfully, the frame extraction algorithm is judged whether it can provide normal service based on the failure log. If the frame extraction algorithm can provide normal service, the network problem / stream problem will be used as the fault location result. If the frame extraction algorithm service is abnormal, the AI ​​middle station problem will be used as the fault location result.

4. The method according to claim 1, characterized in that: The method further comprises: According to a preset plurality of monitoring indicators and a preset collection cycle, the indicator data of the AI ​​visual analysis service is collected; wherein the plurality of monitoring indicators include service quality indicators and business indicators; According to the preset inspection cycle, automatic inspection is performed according to the indicator data, and the automatic inspection result is obtained; the automatic inspection result is normal or abnormal.

5. The method according to claim 4, characterized in that The collecting of indicator data of the AI ​​visual analysis service includes: A plurality of probe collectors are used to respectively collect the indicator data of the AI ​​visual analysis service; wherein the plurality of probe collectors are configured by reusing the scrape_configs configuration item of the Promethus tool.

6. The method according to claim 4, characterized in that The automatic inspection is performed according to the indicator data, and the automatic inspection result is obtained, including: Performing automated inspections based on the indicator data; If at least one target alarm data is generated during the automated inspection process, historical alarm data of the target AI visual analysis service is obtained, and the at least one target alarm data is filtered based on the historical alarm data to obtain a filtering result; If the filtering result is empty, the automatic inspection result is normal, otherwise, the automatic inspection result is abnormal.

7. The method according to claim 4, characterized in that The automatic inspection is performed according to the indicator data, and the automatic inspection result is obtained, including: Performing automated inspections based on the indicator data; If at least one target alarm data is generated during the automated inspection process, the at least one target alarm data is subjected to noise reduction filtering according to the alarm noise reduction rule configured in advance through the low-code platform to obtain a filtering result; If the filtering result is empty, the automatic inspection result is normal, otherwise, the automatic inspection result is abnormal.

8. The method according to claim 6 or 7, characterized in that: Generating and pushing a fault troubleshooting result according to the fault location result includes: If the filtering result is not empty, the target alarm data in the filtering result is pushed to the alarm pushing user according to the alarm pushing rule customized by the alarm pushing user; wherein the alarm pushing rule includes at least one of the following: alarm level, notification method and callback interface.

9. The method according to claim 4, characterized in that The indicator data includes an indicator value, a device identifier of an indicator source device, and an algorithm code of the target AI visual analysis service; the method further includes: Constructing a visual dependency graph based on the device identification and algorithm coding of each of the indicator data; Based on the visualized dependency graph and each of the indicator values, a visualized fault diagnosis path is generated.

10. An automated monitoring and operation device for AI visual analysis services, characterized in that: include: A judgment module, used to judge whether the AI ​​middle station generates target alarm data for a target AI visual analysis service in response to the fault troubleshooting interface being called or the automatic inspection result being abnormal; wherein the target AI visual analysis service is an AI visual analysis service that needs to be troubleshooted; The first fault location module is used to locate the fault according to the subscription information and push log of the target AI visual analysis service if the AI ​​middle station generates the target alarm data, and obtain the fault location result; A second fault location module is used to locate the fault according to the task details of the AI ​​middle station, the target device status and the target algorithm status corresponding to the target AI visual analysis service if the AI ​​middle station does not generate the target alarm data, and obtain the fault location result; wherein the target device status is the status of the device used to run the target AI visual analysis service; The push module is used to generate and push the fault troubleshooting result according to the fault location result.

Images