An AI visual analysis service automatic monitoring operation and maintenance method and related device

Abstract

The application provides an automatic monitoring and operation method of an AI visual analysis service and related devices. By providing an interface for forcibly starting a troubleshooting process, when no alarm is generated by automatic inspection, an operator or technician can call the interface to forcibly start the automatic troubleshooting process of analyzing faults, and return the troubleshooting results in a synchronous or asynchronous manner. In this way, the operator or developer does not need to manually log in to the system environment and manually confirm one by one according to the automatic inspection process, thereby improving fault handling efficiency and fault handling accuracy. According to whether the target alarm data is generated in the AI middle platform, the application selects the subscription information and push logs of the target AI visual analysis service for fault positioning, or selects the task details, target device status and target algorithm status corresponding to the target AI visual analysis service of the AI middle platform for fault positioning, thereby obtaining accurate fault positioning results and further improving the accuracy of fault handling.

Description

Technical Field

[0001] This application relates to the field of monitoring and maintenance technology, and in particular to an automated monitoring and maintenance method and related apparatus for AI visual analysis services. Background Technology

[0002] Currently, AI vision technology has been widely applied across various industries, and numerous AI video analytics systems have emerged that integrate computing power, algorithms, and platforms to uniformly manage and automate the training of AI vision algorithms. These AI video analytics systems provide over 500 million AI vision analysis computations daily, therefore, their system stability and reliability are crucial to the service quality of AI vision analytics.

[0003] In order to improve system stability and reliability, in addition to continuously optimizing system architecture and system performance, reducing the frequency of failures, narrowing the scope of failure impact, and reducing failure recovery time are important means to improve system stability and reliability. This requires timely detection and handling of system failures during system monitoring and maintenance.

[0004] Currently, some existing implementation methods rely on professional technicians for fault analysis, troubleshooting, and handling. Take, for example, the frequently complained-about issue of AI vision atomic capability response timeouts and the prolonged delays in receiving algorithm analysis and detection results from third-party platforms after subscribing to AI vision atomic capabilities. When these problems occur, system maintenance and management often depend on professional technicians to analyze the faults and determine whether the problem is caused by factors such as interface service call response timeouts, network failures leading to abnormal video stream parsing by the AI ​​vision algorithm, or faults in the AI ​​vision algorithm service itself. However, considering the diverse causes of faults, manual troubleshooting and handling would result in high maintenance costs, low efficiency, and a high risk of errors. Furthermore, with the continuous expansion of business scale and the increasing complexity of systems, increasingly higher demands are placed on system stability and reliability, making manual monitoring and maintenance methods insufficient to meet the practical needs of early detection, accurate location, and rapid delivery.

[0005] To address the aforementioned issues, existing technologies propose an automated operation and maintenance method for centralized product monitoring. This method achieves centralized monitoring and automated operation and maintenance by setting product monitoring points, uniformly processing monitoring data, visually displaying processing results on a monitoring platform, intelligently analyzing faults, and sending data to the automated operation and maintenance terminal. This approach improves monitoring accuracy and timeliness, reduces labor costs, increases operation and maintenance efficiency, and enables intelligent analysis and alarms for product faults. Furthermore, this method helps R&D personnel better manage and monitor products, improving their stability and reliability.

[0006] However, existing automated monitoring and maintenance solutions still fall short in terms of efficiency and accuracy in fault handling. Summary of the Invention

[0007] The purpose of this application is to at least solve one of the aforementioned technical defects, particularly the technical defects of low fault handling efficiency and low fault handling accuracy in the prior art.

[0008] In a first aspect, embodiments of this application provide an automated monitoring and maintenance method for AI visual analytics services, including:

[0009] In response to the fault diagnosis interface being called or the automatic inspection result being abnormal, it is determined whether the AI ​​platform generates target alarm data for the target AI visual analysis business; wherein, the target AI visual analysis business is the AI ​​visual analysis business that needs to be diagnosed.

[0010] If the AI ​​platform generates the target alarm data, then the fault location is performed based on the subscription information and push logs of the target AI visual analysis service, and the fault location result is obtained.

[0011] If the AI ​​platform does not generate the target alarm data, then the fault location is performed based on the task details of the AI ​​platform, the target device status, and the target algorithm status corresponding to the target AI visual analysis service, and the fault location result is obtained; wherein, the target device status is the status of the device used to run the target AI visual analysis service;

[0012] Based on the fault location results, a fault investigation result is generated and pushed out.

[0013] Secondly, embodiments of this application provide an automated monitoring and maintenance device for AI visual analysis services, comprising:

[0014] The judgment module is used to determine whether the AI ​​platform generates target alarm data for the target AI visual analysis business in response to the fault diagnosis interface being called or the automatic inspection result being abnormal; wherein, the target AI visual analysis business is the AI ​​visual analysis business that needs to be diagnosed.

[0015] The first fault location module is used to locate the fault based on the subscription information and push logs of the target AI visual analysis service if the AI ​​platform generates the target alarm data, and obtain the fault location result.

[0016] The second fault location module is used to locate the fault based on the task details of the AI ​​platform, the status of the target device, and the status of the target algorithm corresponding to the target AI visual analysis service if the AI ​​platform does not generate the target alarm data, and to obtain the fault location result; wherein, the status of the target device is the status of the device used to run the target AI visual analysis service;

[0017] The push module is used to generate and push fault investigation results based on the fault location results.

[0018] In an automated monitoring and maintenance method and related apparatus for AI visual analysis services provided in some embodiments of this application, an interface for forcibly initiating a fault diagnosis process is provided. When the automated inspection does not generate an alarm, maintenance personnel or technicians can call this interface to forcibly initiate the automated fault diagnosis process and receive the fault diagnosis results returned synchronously or asynchronously. This eliminates the need for maintenance or R&D personnel to manually log into the system environment and manually confirm each step according to the automated inspection process, thereby improving fault handling efficiency and accuracy.

[0019] Meanwhile, this application can use the subscription information and push logs of the target AI visual analysis service for fault location based on whether the AI ​​platform generates target alarm data, or use the task details of the AI ​​platform, the status of the target device and the status of the target algorithm corresponding to the target AI visual analysis service for fault location, thereby obtaining accurate fault location results and further improving the accuracy of fault handling. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts illustrating an automated monitoring and maintenance method for AI visual analytics services in one embodiment.

[0022] Figure 2 This is one of the schematic diagrams of a troubleshooting process in one embodiment;

[0023] Figure 3A This is one of the schematic diagrams of monitoring metrics in one embodiment;

[0024] Figure 3B This is a second schematic diagram of the monitoring metrics in one embodiment;

[0025] Figure 4 This is a second schematic diagram of a troubleshooting process in one embodiment;

[0026] Figure 5 As one embodiment, a visual troubleshooting flowchart is provided.

[0027] Figure 6 This is a schematic diagram of the structure of an automated monitoring and maintenance device for AI visual analysis services in one embodiment;

[0028] Figure 7 This is a diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The following is an explanation of the terms used in this application:

[0031] AI (Artificial Intelligence) visual analysis: using artificial intelligence technology to automatically analyze and understand images or videos, extract useful information, and achieve functions such as target detection, tracking, and recognition.

[0032] Low-code platform: A software development platform that allows users to quickly build applications with a graphical interface and a small amount of code, reducing the development threshold and cycle time, improving development efficiency, and quickly producing applications, thereby accelerating time to market.

[0033] Automated and precise monitoring: Through preset alarm rules, standard fault handling procedures and algorithms, the system or business is automatically monitored in real time and faults are self-healed, ensuring timely detection and handling of abnormal situations and achieving precise operation and maintenance.

[0034] As mentioned in the background section, existing technologies still fall short in terms of efficiency and accuracy in fault handling. The inventors' research revealed that this problem stems from the following: when automated business inspections fail to generate alarms, if operations or front-end personnel receive complaints from customers / applications regarding designated tasks that should have generated alarms but did not, maintenance or development personnel often need to manually log into the system environment and manually verify each step according to the automated inspection process. This significantly impacts the efficiency and accuracy of fault handling.

[0035] To address the aforementioned issues, this application provides an interface for directly querying specified faults. Users can call this interface to force the start of the fault analysis and troubleshooting process and receive the fault troubleshooting results returned synchronously or asynchronously.

[0036] In addition to the technical problems mentioned above, existing technologies have at least the following four problems:

[0037] (1) Existing technologies first set up product monitoring points and then centrally monitor these points based on preset monitoring rules to obtain monitoring data. In this process, existing technologies mainly monitor service quality indicators (such as health status indicators of middleware, databases, virtualization, network devices, servers, host hardware, etc.) through product monitoring points, ignoring the monitoring of business indicators. However, normal service quality indicators do not necessarily mean that business indicators are also normal. For example, when there is a network failure from the client to the server, service quality indicators such as the server's success rate and latency may be normal, but for the client, it will be unable to subscribe to AI vision services, unable to call AI vision services for video analysis, and unable to receive the algorithm results pushed by AI vision services.

[0038] Although there are many alarm configuration strategies for cause-related indicators, such as machine crash alarms, high CPU utilization alarms, and high memory utilization alarms, the lack of monitoring of business-related indicators makes it impossible to clearly identify the impact of cause-related indicators on the business level. This lack of specificity leads to a high false alarm rate and makes it impossible to ensure that every fault is reported.

[0039] (2) In the existing technology, there are many types of monitoring and data collection tools relied upon by product monitoring points. This not only leads to inconsistencies in the user experience but also increases the learning cost for technical personnel. In addition, the configuration process of monitoring and alarm strategies is complex and cumbersome, and the maintenance work is heavy. These all require personnel with professional skills to undertake, which increases the cost of monitoring operation and maintenance. The high cost of monitoring operation and maintenance and delivery makes it difficult for the entire system to flexibly adapt to rapidly changing business needs.

[0040] (3) Data silos are significant, requiring maintenance personnel to check each independent monitoring data source one by one. This process is not only costly and inefficient in terms of correlation analysis, but also greatly increases the complexity and time consumption of troubleshooting. This decentralized data management approach often leads to untimely anomaly handling, making it difficult to effectively ensure the stable operation of the platform, and even more difficult to achieve the goals of early fault detection, accurate location, and rapid loss mitigation.

[0041] (4) The operations and maintenance team may face hundreds or even more alarms every day. In many cases, only a few of them are valid and need to be dealt with immediately. However, these critical alarms are often overwhelmed by a large number of false alarms, making it difficult to identify and deal with them in a timely manner. What is more troublesome is that because different roles such as operations, R&D, and customers have different concerns about system failures, it is difficult for each position to respond to and deal with the real problems in the first place.

[0042] In addition, while existing alarm systems can basically achieve fault notification functions and describe the cause and severity of faults through preset methods, they lack in-depth analysis and are unable to provide key information such as the scope of fault impact, root cause, and repair suggestions. This makes it difficult for maintenance and R&D personnel to judge the effectiveness of automatic repair measures, and they often need to manually log into the system environment to confirm the repair effect one by one, which greatly affects the efficiency and accuracy of fault handling.

[0043] To address the aforementioned issues, in some embodiments, this application integrates service quality target monitoring indicators and AI vision business monitoring indicators in the automated monitoring and maintenance of AI video services. This constructs a comprehensive business monitoring system, enabling flexible configuration and customized monitoring, simplified management of unified monitoring probes, intelligent detection and early warning, efficient fault handling and location, and in-depth analysis and automatic repair. These measures significantly improve the operational efficiency and system stability of AI vision services, demonstrating strong practicality. The automated monitoring and maintenance solution for AI vision analysis services provided by this application includes at least one of the following five aspects:

[0044] (1) Adopt a combination of indicators for monitoring service quality targets and indicators for AI vision analysis business, attach importance to the construction and monitoring of business indicator system, enumerate and list the result indicators and fault categories corresponding to the core business functions, make good monitoring rules for result indicators, and build a relatively comprehensive business monitoring indicator system. This system is conducive to the efficient operation of AI vision business and the continuous optimization of products.

[0045] (2) Integrate all detection and acquisition tools related to AI vision business analysis and encapsulate them into a unified monitoring probe collector. This collector can comprehensively cover all monitoring targets, achieve centralized management, simplify the configuration process, and allow users to directly specify target details and acquisition rules in the configuration of the monitoring probe collector, thereby automating the collection, cleaning, and classification of monitoring data. In addition, by developing standardized templates for configuration files, not only are the problems of inconsistent configuration file formats and varying user experiences between different plugins eliminated, but the learning threshold for the technical team is also significantly reduced, promoting rapid deployment and delivery of projects and better adapting to rapid changes in market and business needs. The unified monitoring probe collector can not only integrate third-party plugins, but also extend to custom monitoring targets. It can obtain data from business database forms, generate key business indicators based on preset algorithms, such as the usage of core business function services and the volume of AI visual alarm pushes. According to the pre-established data specifications, it is transformed into a unified format, and these data are linked together to form business indicators. Combined with business needs, it performs intelligent anomaly detection on the indicators, discovers abnormal trends in business indicators, and issues alarms in a timely manner, truly turning passive into proactive and discovering faults before complaints are received.

[0046] (3) By quickly building and changing alarm notification methods, alarm strategies, and visual monitoring templates through a low-code platform, the configuration of monitoring rules is simplified, the technical threshold is lowered, and maintenance costs are reduced. When a business is about to fail or there is a potential fault, it can not only be proactively detected and warned by the first monitoring event, but also automatically perform in-depth analysis and accurately describe the method to eliminate the potential fault through the warning notification. For proactively detected faults, it can not only automatically perform preliminary location and accurately narrow down and describe the scope of the fault point through alarm notification, but also automatically perform in-depth analysis and accurately and in detail describe the cause of the fault and the method to repair the fault through the alarm. While transmitting the data to be transmitted to the alarm terminal in the form of an alarm, the fault can be automatically repaired under certain conditions. Moreover, reasonable configuration of alarm noise reduction and alarm suppression strategies can intuitively and efficiently perform closed-loop tracking of troubleshooting these alarm problems.

[0047] (4) Based on different user roles (operations, maintenance, or R&D personnel), users can subscribe to the monitoring rules they want to receive, subscribe to unified and standardized monitoring rules, and redefine alarm levels, notification methods, and callback interfaces, thereby improving alarm response speed. This better adapts to the ever-changing market environment and accelerates the promotion of AI visual analysis services.

[0048] (5) The monitoring and early warning system is combined with in-depth analysis. In step 3, the alarm notification not only automatically provides methods for eliminating potential faults, but also includes fault location, cause analysis, and repair guidance. Since the task of in-depth analysis of business faults is essentially an automatic inspection followed by detailed analysis, there will inevitably be a certain time delay. Users can force the start of a custom in-depth analysis process for troubleshooting business faults, and return the results of the specified troubleshooting (e.g., fault location, cause analysis, and repair guidance) in a synchronous or asynchronous manner.

[0049] The following describes the automated monitoring and maintenance method for the AI ​​visual analysis service provided in this application.

[0050] In some embodiments, such as Figure 1 As shown, this application provides an automated monitoring and maintenance method for AI visual analytics services, including the following steps:

[0051] S102: In response to the fault diagnosis interface being called or the automatic inspection result being abnormal, determine whether the AI ​​platform generates target alarm data for the target AI visual analysis business; where the target AI visual analysis business is the AI ​​visual analysis business that needs to be diagnosed.

[0052] S104: If the AI ​​platform generates target alarm data, then the fault location is performed based on the target AI visual analysis business subscription information and push logs, and the fault location result is obtained.

[0053] S106: If the AI ​​platform does not generate target alarm data, then the fault location is performed based on the task details of the AI ​​platform, the target device status, and the target algorithm status corresponding to the target AI visual analysis service, and the fault location result is obtained; where the target device status is the status of the device used to run the target AI visual analysis service;

[0054] S108: Generate and push fault diagnosis results based on fault location results.

[0055] Among them, the fault diagnosis interface is used to force the start of the fault diagnosis process. Operation and maintenance personnel or technicians can start the fault diagnosis process for the target AI visual analysis business by calling the fault diagnosis interface.

[0056] This application provides an automatic inspection function and an interface for forcibly initiating a fault diagnosis process. When the automatic inspection result is abnormal, a fault diagnosis process can be initiated to troubleshoot the target AI visual analysis business. This application can also initiate a fault diagnosis process when the fault diagnosis interface is called. It should be noted that when the fault diagnosis interface is called, regardless of whether the automatic inspection structure generates an alarm, this application must forcibly execute the fault diagnosis process to deeply analyze the business fault and return the fault diagnosis results.

[0057] For example, when automatic inspections fail to generate alarms, but operations or front-end personnel receive a report from a customer or application side regarding a specified task that should have generated alarms but did not, they can invoke the fault investigation interface to forcibly initiate a custom in-depth analysis of business faults and return the investigation results. This method is mainly for items with long automatic inspection intervals or long execution frequencies and durations set in the intelligent noise reduction process, as this may prevent the inspection results from transmitting the detected faults to the alarm terminal in a timely manner. Therefore, when operations or front-end personnel receive complaints from customers or applications, after adding a specified task that requires in-depth analysis of business fault location results, the custom monitoring business process will forcibly initiate the in-depth analysis of business fault investigation logic, returning the specified fault investigation results synchronously or asynchronously.

[0058] During the troubleshooting process, this application can first check the alarm data of the AI ​​platform to determine whether the AI ​​platform has generated target alarm data. Target alarm data refers to alarm data specific to the target AI visual analysis service. If the AI ​​platform has generated at least one target alarm data, the fault location can be determined based on the subscription information and push logs of the target AI visual analysis service.

[0059] If the AI ​​platform does not generate target alarm data, the task details of the AI ​​platform, the status of the device used to run the target AI visual analysis service (i.e., target device status), and the task status of the algorithm side (i.e., target algorithm status) can be viewed separately. The task details of the AI ​​platform reflect the task status on the platform side, and the target algorithm status reflects the task status on the algorithm side. In the absence of target alarm data generated by the AI ​​platform, this application can perform fault location based on the task details, target device status, and target algorithm status, and determine the fault location result.

[0060] After determining the fault location result, this application can generate a fault diagnosis result based on the fault location result and push the fault diagnosis result to relevant personnel. In some embodiments, the fault diagnosis result may include the fault location result, fault cause analysis, and repair guidance suggestions.

[0061] Furthermore, where conditions permit, this application can activate an automatic fault repair function, such as by restarting the service, switching to a backup device, or horizontal scaling, to achieve an automated processing mechanism, thereby further reducing manual intervention. In some examples, this application can use one of the following two methods to implement the automatic fault repair function:

[0062] (1) The troubleshooting results are pushed to an external system via Webhook, and the external system can perform corresponding actions based on the content of the troubleshooting results;

[0063] (2) Distribute the repair script to the alarm device (i.e., the faulty device) or the specified detection target point so that the alarm device can execute the repair script to achieve fault repair. The repair script can be a shell script, a python script, a bat script, etc.

[0064] To automatically trigger the execution of repair scripts on alarm devices, a probe needs to be installed at the detection target point. This probe periodically sends heartbeats to the server, retrieves the script to be executed, and reports the script execution results to the server. If automatic fault repair is enabled, when an alarm event occurs, not only will the details of the fault repair script be displayed, but the server will also create an execution task, waiting for the probe to retrieve and execute it. When fault repair notification is enabled, if the fault recovery policy is met, the recovered monitoring items, the stdout of the automatically triggered repair script execution, and the measurement values ​​of the monitoring data will be transmitted to the alarm terminal to indicate that the fault has been repaired.

[0065] Typically, one alarm corresponds to one remediation script, or the remediation script can be made more general to handle different scenarios. In some examples, key information about the alarm event (such as the detection target point, alarm policy name, etc.) can be transmitted to the remediation script via standard input stdin. Upon receiving the alarm event content, the remediation script can then perform different processing steps.

[0066] Furthermore, in the event of an impending or potential business failure, this application can proactively detect and issue early warnings, and automatically perform in-depth analysis, accurately describing methods to eliminate potential failures through early warning notifications. For proactively detected failures, it can not only automatically perform initial location analysis, accurately narrowing down the scope of the failure point through alarm notifications, but also automatically perform in-depth analysis, accurately and comprehensively describing the cause of the failure and the method of repair through alarms. While transmitting the data to be transmitted to the alarm terminal via alarms, it can automatically complete the fault repair under certain conditions. When an alarm event occurs, it will not only display the details of the script used for fault repair, but the server will also create an execution task, waiting for the probe to pull and execute it. When fault repair notifications are enabled, if the fault recovery strategy is met, it will transmit the recovered monitoring items, the standard output (stdout) of the automatically triggered repair script execution, and the measurement values ​​of the monitoring data to the alarm terminal, informing that the fault has been repaired. This allows for intuitive and efficient closed-loop tracking of troubleshooting these alarm issues.

[0067] In some embodiments, such as Figure 2 As shown, fault location is performed based on the subscription information and push logs of the target AI visual analysis service, and the fault location results are obtained, including:

[0068] Based on subscription information and push logs, determine whether the target AI visual analysis service has been subscribed to and whether there are any push anomalies.

[0069] If the target AI visual analysis service has no subscription or has abnormal push notifications, the application problem will be used as the fault location result.

[0070] If the target AI visual analysis service has subscriptions and the push logs are empty, then the AI ​​middleware push service anomaly will be used as the fault location result.

[0071] The subscription information for the target AI visual analytics service may include subscriber information and / or information reflecting whether the target AI visual analytics service has been subscribed to. The push logs for the target AI visual analytics service can be used to record information about messages pushed to subscribers.

[0072] Specifically, if the AI ​​platform has target alarm data, the subscription information and push logs of the target AI visual analysis service can be viewed separately. If the target AI visual analysis service has no subscription or has push anomalies, it can be determined that the fault is on the application side, and therefore the application problem can be used as the fault location result. In one example, after determining that it is an application problem, this application can report the relevant fault to the application.

[0073] If the target AI visual analysis service has a subscription but no corresponding message push, it indicates that the application has received the subscription but cannot complete the message push. Therefore, it can be determined that there is an anomaly in the AI ​​middleware's push service, and this anomaly can be used as a fault location result. Furthermore, after determining that the problem is with the AI ​​middleware's push service, this application can investigate the AI ​​middleware's microservices for more in-depth analysis, thereby accurately and thoroughly identifying the cause of the fault.

[0074] This can further improve the accuracy of fault location results, thereby further improving the accuracy of fault handling.

[0075] In some embodiments, task details include the target task status generated by the AI ​​platform for the target AI visual analysis business, i.e., the task status on the platform side.

[0076] like Figure 2 As shown, fault location is performed based on the task details of the AI ​​platform, the status of the target device, and the status of the target algorithm corresponding to the target AI visual analysis business, and the fault location results are obtained, including:

[0077] If the target task status is normal and the target algorithm status is normal, then if the business type of the target AI visual analysis business is video business, the algorithm-side problem will be used as the fault location result.

[0078] If the target task status is normal and the target algorithm status is normal, then if the business type is frame extraction business and there are call logs, the algorithm-side problem will be taken as the fault location result. If the business type is frame extraction business and there are no call logs, the algorithm pulling and frame extraction problem will be taken as the fault location result.

[0079] If the target task status is abnormal, and the target algorithm status is push abnormal / the target device status is offline, then attempt to perform a capture operation on the video stream involved in the target AI visual analysis business and obtain the video stream address;

[0080] If the capture is successful, the algorithm's frame-pulling problem will be used as the fault location result.

[0081] If the capture fails and the video stream address fails to be obtained, the video aggregation platform problem / equipment problem will be used as the fault location result.

[0082] If the capture fails but the video stream address is successfully obtained, the failure log is used to determine whether the frame extraction algorithm can provide normal service. If the frame extraction algorithm can provide normal service, the network problem / stream problem is used as the fault location result. If the frame extraction algorithm service is abnormal, the AI ​​platform problem is used as the fault location result.

[0083] Specifically, when the AI ​​platform has no target alarm data, this application can view the task details of the AI ​​platform to determine the task status of the target AI analysis service on the platform side. This application can also view the target algorithm type corresponding to the target AI visual analysis service to determine the task status on the algorithm side. In addition, this application also views the device status.

[0084] When both the target task and the target algorithm are in normal status, it indicates that both the platform and the algorithm consider the target AI visual analysis service to be executing normally. In this case, analysis can be performed based on the service type of the target AI visual analysis service. If the service type of the target AI visual analysis service is video service, then the problem is determined to be on the algorithm side, and the algorithm side problem can be used as the fault location result. Furthermore, this application can investigate the algorithm or network connection to analyze the cause of the fault in depth.

[0085] When both the target task and the target algorithm are functioning normally, if the target AI visual analysis service is a frame extraction service, the call logs of the target AI visual analysis service can be further examined. If the target AI visual analysis service has call logs, the problem is on the algorithm side. Further investigation can be conducted to examine the algorithm or network connection for a more in-depth analysis of the cause of the failure. If there are no call logs, it indicates that the target AI visual analysis service failed to successfully pull the stream for frame extraction; therefore, the problem is with the algorithm's streaming and frame extraction. Further investigation can be conducted to examine the frame extraction algorithm or network for a more in-depth analysis of the cause of the failure.

[0086] If the target task status is abnormal and the target algorithm status is push abnormal, then attempt to perform a snapshot operation on the video stream involved in the target AI visual analysis service and obtain the video stream address. If the target task status is abnormal and the target device status is offline, then attempt to perform a snapshot operation on the video stream involved in the target AI visual analysis service and obtain the video stream address.

[0087] If the video stream involved in the target AI visual analysis service is successfully captured, the problem can be identified as an algorithm-related frame-dropping issue, and this can be used as the fault location result. If the capture fails, further analysis can be performed based on the video stream address acquisition result. If the video stream address acquisition fails, the problem can be identified as a video aggregation platform / device issue, and this can also be used as the fault location result. Furthermore, feedback can be provided to the video aggregation platform for inspection and in-depth analysis.

[0088] If the capture fails but the video stream address is successfully obtained, the failure log is used to determine whether the frame extraction algorithm can provide normal service. If the frame extraction algorithm is working normally, the problem is likely a network or streaming issue. If the frame extraction algorithm fails, the problem is likely on the AI ​​platform side. Furthermore, if the frame extraction algorithm fails, this application can restore the frame extraction algorithm used for testing purposes.

[0089] Furthermore, if the target task status is normal and the target algorithm status is "task does not exist," then this application can operate to restore the algorithm task. If restoring the algorithm task fails, it can be determined that the problem lies with the AI ​​platform.

[0090] This can further improve the accuracy of fault location results, thereby further improving the accuracy of fault handling.

[0091] In some embodiments, the automated monitoring and maintenance method for AI visual analysis services provided in this application further includes the following steps:

[0092] Based on multiple preset monitoring indicators and preset collection cycles, the system collects indicator data for AI visual analysis services; among these, the multiple monitoring indicators include service quality indicators and business indicators.

[0093] The system performs automated inspections based on indicator data according to a preset inspection cycle, and obtains the results of the automated inspections, which are either normal or abnormal.

[0094] Specifically, this application can combine metrics for monitoring service quality objectives with metrics for AI visual analysis services, emphasizing the construction and monitoring of a business metric system. It enumerates and lists the corresponding result-type metrics and fault categories for core business functions, thereby obtaining multiple monitoring metrics. In one example, the various monitoring metrics of this application can be as follows: Figure 3A and Figure 3B As shown, Figure 3A and Figure 3B The data includes business metrics and service quality metrics.

[0095] Similar to common IT architectures, the server-side architecture for AI visual analytics generally includes the following four layers:

[0096] (1) Functional interfaces: such as login interface, subscription interface, alarm push interface, etc., usually numbering in the hundreds or thousands;

[0097] (2) Microservices: such as user management programs, AI vision manufacturer management programs, AI vision analysis result summary and display programs, alarm data push programs, etc., which are developed, written and run program services;

[0098] (3) Standard middleware: such as MySQL, Redis, Kafka, etc.

[0099] (4) Infrastructure: such as switches, cloud servers, video equipment, etc.

[0100] All objects are hierarchically grouped, and the health metrics of each type of object are standardized. Users are guided to analyze the indicators, logs, links, and events of abnormal objects and combine them into visual monitoring to facilitate the analysis and confirmation of faults.

[0101] With multiple monitoring indicators set, this application can collect indicator data of AI visual analysis business according to a preset collection cycle. The collected indicator data includes indicator data corresponding to service quality indicators and indicator data corresponding to business indicators.

[0102] This application can automatically inspect the target AI visual analysis business based on the collected indicator data, according to the pre-set inspection cycle and preset alarm rules, in order to determine whether there are alarms in the target AI visual analysis business and obtain the automated inspection results.

[0103] In one example, consider a tenant who has subscribed to the AI ​​vision analytics service reporting that they have not received the detection and analysis results of the subscribed AI vision atomic capabilities. Briefly explain the specification of alarm rules. Please refer to... Figure 4 and Figure 5 This application can automatically inspect all tasks of the AI ​​vision business service at regular intervals. Only when a task falls under one of the following conditions is its alarm status considered normal, and the metric value is marked as "0", without initiating a fault investigation process or alarm notification:

[0104] (1) The task has no corresponding subscription information, and the alarm data on the platform side has been successfully pushed to the view library;

[0105] (2) The task has corresponding subscription information, and the subscription address contains a string containing the internal whitelist (this indicates that the application side obtains the alarm data by subscribing to the view library), and the alarm data on the platform side is successfully pushed to the view library.

[0106] (3) The task has corresponding subscription information, and the subscription address does not contain strings from the internal whitelist (this indicates that the application side obtains alarm data through the subscription platform), and the alarm data from the platform side is successfully pushed to the address.

[0107] (4) For video tasks, if no alarm data for that task is found on the platform for that day, but the task status on the platform is normal, and the task exists and is in a normal status within the test query algorithm container, then according to the location of the investigation layer, the problem may be with the image algorithm. In this case, the current time point can be determined. If the current time point has not exceeded the preset time (e.g., 12 noon), it indicates that the time since the last inspection (or the time when the task was put into use) has not yet reached a complete time cycle (e.g., one day), so no alarm can be issued for the time being.

[0108] If the current time exceeds a preset time, the total number of alarms for the task in the previous complete time period (hereinafter referred to as the first total number) can be queried and compared with the total number of alarms in the current complete time period (hereinafter referred to as the second total number). When the difference between the first total number and the second total number is greater than a preset threshold, it is determined that the task is abnormal and an alarm is issued; otherwise, no alarm is issued for the time being.

[0109] (5) For frame extraction and arrangement tasks, no alarm data for the task on the platform was found for that day, but the task status on the platform side was normal. The task in the algorithm service was found to exist and be in normal status. In addition, there were logs of calling image algorithms. Based on the location of the troubleshooting process, the problem may be related to the image algorithm. In this case, the current time point can be determined. If the current time point has not exceeded the preset time (e.g., 12 noon), it means that the time since the last inspection (or the time when the task was put into use) has not yet reached a complete time cycle (e.g., one day). Therefore, no alarm can be issued for the time being.

[0110] If the current time exceeds a preset time, the total number of alarms for the task in the previous complete time period (hereinafter referred to as the third total number) can be queried and compared with the total number of alarms in the current complete time period (hereinafter referred to as the fourth total number). When the difference between the third total number and the fourth total number is greater than a preset threshold, it is determined that the task is abnormal and an alarm is issued; otherwise, no alarm is issued for the time being.

[0111] If none of the above conditions are met, the metric value is marked as "1," requiring the initiation of a fault diagnosis process. This process automatically performs a more in-depth analysis to accurately and thoroughly identify the cause of the fault. The analyzed fault cause and the fault repair method can be transmitted to the alarm terminal via an alarm. For a detailed description of the fault diagnosis process, please refer to the above embodiments; it will not be repeated here.

[0112] In some embodiments, the collection of metric data for AI visual analytics services includes:

[0113] Multiple probe collectors are used to collect metric data for AI visual analysis services; the multiple probe collectors are configured using the scrape_configs option of the Promethus tool.

[0114] In this embodiment, considering the diversity of monitoring indicators, multiple probe collectors are required to collect indicator data. However, in the field of monitoring data collection, there are many native probe collectors. Although the ecosystem is thriving, they are not ready to use out of the box. Moreover, the log libraries, configuration file management methods, and command line parameter passing methods used by probe collectors vary, resulting in inconsistent user experiences. Therefore, this application integrates all detection and collection tools related to AI vision business analysis and encapsulates them into a unified monitoring probe collector. This collector can comprehensively cover all monitoring targets, achieve centralized management, simplify the configuration process, and allow users to directly specify target details and collection rules in the configuration of the monitoring probe collector, thereby automating the collection, cleaning, and classification of monitoring data.

[0115] Furthermore, by establishing standardized templates for configuration files, not only are issues of inconsistent configuration file formats and varying user experiences between different plugins eliminated, but the learning curve for technical teams is also significantly lowered, facilitating rapid project deployment and delivery, and better adapting to rapid changes in market and business needs. The unified monitoring probe collector can not only integrate third-party plugins but also extend to custom monitoring targets. It retrieves data from business database forms, generates key business metrics based on preset algorithms, such as the usage of core business function services and the volume of AI visual alarm pushes. According to pre-established data specifications, it is transformed into a unified format, and these data are linked together to form business metrics. Combined with business needs, intelligent anomaly detection is performed on these metrics, identifying abnormal trends and issuing timely alerts, truly turning passive responses into proactive ones, discovering faults before complaints are received.

[0116] Specifically, this application modifies the configuration items of the unified monitoring probe collector based on the visualized fault diagnosis flowchart, specifying the information of the target to be crawled, the crawling rules, etc., establishing the standardization of monitoring data, and ensuring the cleaning, classification, and persistent storage of the monitoring data. This configuration reuses Promethus's scrape_configs as much as possible. Each scrape_config contains parameters such as job name, target address, crawling interval, and timeout, thereby reducing the learning cost. Furthermore, scrape_configs can be extended. For example, when monitoring the metrics of a specific table in a specific database, some SQL statements need to be configured; these are the crawling rules. These crawling rules are referenced in the scrape_configs, and managed using a file-based approach, allowing different crawling rules to be used for different detection targets, providing great flexibility.

[0117] Furthermore, all configurations are placed in a single directory file. Each subdirectory within this file represents a type of plugin. For example, a `kafka` directory within the configuration directory contains configurations related to the Kafka data collection plugin, and an `elasticsearch` directory contains configurations related to the Elasticsearch data collection plugin. Each plugin directory contains one or more entry configuration files. When the monitoring probe collector matches an entry configuration file, it sequentially reads and assembles the data into a single configuration file based on the configured collection targets and rules within each file before capturing the data. After the unified monitoring probe collector collects the data, it is ultimately pushed to a remote time-series database via the Prometheus RemoteWrite protocol. Multiple push addresses can be configured. If a backend only wants to receive specific data, it can use relabeling to reset the label configuration for filtering. Different push addresses can be appended with different labels and can also be configured with different relabeling rules.

[0118] The primary purpose of the unified probe collector is to manage all monitoring metrics, standardize the inconsistent configuration styles of different monitoring points, and ensure out-of-the-box usability. It acquires key metrics from the monitoring points, encapsulates them into time-series database metrics, and then the unified monitoring probe collector writes these data to a remote time-series database for persistent storage.

[0119] Furthermore, this application not only integrates third-party plugins but also extends custom monitoring metrics. The specific example listed above involves retrieving data from business database forms and calculating business metrics. Custom monitoring metrics can also include the usage of core business function interfaces and the volume of AI visual analysis results pushed to users. Based on pre-established data specifications, such as defining log recording targets, log levels, and ensuring that applications and services use a consistent and structured log format (unstructured logs are transformed into a unified format), these data are linked together. Detailed contextual information related to the logs is recorded and summarized into business metrics. Combined with business needs, intelligent anomaly detection is performed on these metrics. When abnormal trends in business metrics are detected, timely alerts are issued, truly turning passive detection into proactive detection, discovering faults before complaints are received.

[0120] Currently, existing technologies generally offer a poor alarm experience. This is primarily manifested in the fact that operations teams may need to handle hundreds or even more alarms daily, with only a few being valid and requiring immediate attention. These critical alarms are often buried under a massive number of false alarms, making them difficult to identify in a timely manner. Even more problematic is that the differing concerns of operations, development, and customer roles regarding system failures hinder their ability to respond to and address real issues promptly.

[0121] In this application, to avoid false alarms and duplicate alarms, alarm noise reduction rules are set, such as sending only one notification when the same alarm is triggered multiple times within a short period. Historical alarm data is analyzed using algorithms to identify common false alarm patterns and causes, and the alarm noise reduction rules are adjusted accordingly. During actual operation, the alarm noise reduction rules are continuously tested and optimized to ensure they effectively reduce the false alarm rate and duplicate alarm rate.

[0122] In addition, setting reasonable alarm noise reduction and alarm suppression strategies, and aggregating similar alarms together for dispatch, notification and processing, not only improves efficiency, but also reduces alarm fatigue and false alarms.

[0123] Specifically, in some embodiments, automated inspections are performed based on indicator data, and automated inspection results are obtained, including:

[0124] Automated inspections are performed based on indicator data;

[0125] If at least one target alarm data is generated during the automated inspection process, the historical alarm data of the target AI visual analysis business is obtained, and the at least one target alarm data is filtered based on the historical alarm data to obtain the filtering result.

[0126] If the filter result is empty, the automatic inspection result is normal; otherwise, the automatic inspection result is abnormal.

[0127] In this embodiment, during automated inspection, current alarm data can be filtered and selected based on historical alarm data, thereby achieving alarm data noise reduction. This allows for the automatic determination of alarm noise reduction rules, thus improving the level of automation.

[0128] Specifically, this application can utilize historical data to predict the current state of the target AI visual analysis service, and use historical alarm data to predict whether newly arriving alarms need to be filtered out. This method can involve various algorithm models, including but not limited to any one or any combination of similarity models, classification models, prediction models, and other algorithm models.

[0129] The intelligent noise reduction process requires minimal user configuration. Users only need to set alarm categories and priorities, and configure the frequency and duration of execution of statistical formulas using year-on-year, month-on-month, and aggregation operators based on the target indicator data of the monitoring points. Compared to methods that only set thresholds / dynamic thresholds or default rules, this noise reduction mechanism offers greater flexibility and automation. In some examples, the specific implementation steps are as follows:

[0130] Step A1: Alarm Classification and Priority Setting: Classify alarms according to their type and set different priorities based on their severity and urgency to help operations and maintenance personnel better understand and handle alarm information, and prioritize those alarms that have the greatest impact on system stability and business.

[0131] Step A2: Setting the anomaly threshold, execution frequency, and execution duration: By combining year-on-year, month-on-month, and aggregation operators, and by setting reasonable anomaly thresholds, execution frequency, and execution duration for statistical data characteristics, we can avoid interference from short-term fluctuations in the alarm service. Alarms will only be triggered when the indicator exceeds a certain threshold for a certain period of time, thereby reducing false alarms.

[0132] Step A3: Alarm Deduplication and Aggregation: For identical or similar alarms, deduplication or aggregation is performed to retain only one related alarm message or merge them into one alarm message, reducing the interference of duplicate alarms on operation and maintenance personnel, and also reducing the storage and processing pressure of alarm services.

[0133] Step A4: Alarm masking and masking rules: For known irrelevant alarms or alarms that do not need to be processed temporarily, set masking rules to mask them to avoid interfering with the operation and maintenance personnel. Masking rules can be set based on alarm type, source, time and other conditions.

[0134] In some embodiments, automated inspections are performed based on indicator data, and the automated inspection results are obtained, including:

[0135] Automated inspections are performed based on indicator data;

[0136] If at least one target alarm data is generated during the automated inspection process, the at least one target alarm data is filtered for noise reduction according to the alarm noise reduction rules configured in advance through the low-code platform to obtain the filtering result.

[0137] If the filter result is empty, the automatic inspection result is normal; otherwise, the automatic inspection result is abnormal.

[0138] In this embodiment, a low-code platform allows for the rapid construction and modification of alarm notification methods, reasonable configuration of alarm noise reduction and suppression strategies, and visualization monitoring templates. This simplifies monitoring rule configuration, lowers the technical threshold, and reduces maintenance costs. Simultaneously, it allows users to subscribe to monitoring rules relevant to their roles (operations, maintenance, or R&D personnel), subscribe to standardized monitoring rules, and redefine alarm levels, notification methods, and callback interfaces, improving alarm response speed. This better adapts to the ever-changing market environment and accelerates the promotion of AI visual analytics services.

[0139] In some embodiments, generating and pushing fault diagnosis results based on fault location results includes:

[0140] If the filtering result is not empty, the target alarm data in the filtering result will be pushed to the alarm push user according to the alarm push rules defined by the alarm push user; wherein, the alarm push rules shall include at least one of the following: alarm level, notification method and callback interface.

[0141] In this application, when a business is about to experience a failure or has a potential for failure, it is monitored and proactively detected, and an early warning is issued immediately. It can also automatically perform in-depth analysis and accurately describe methods to eliminate the potential failure through early warning notifications. For proactively detected failures, it can automatically perform preliminary location analysis and configure multiple notification methods, such as SMS, email, telephone, and instant messaging software, according to actual needs. This ensures the reliability and timeliness of notification methods, enabling the delivery of alarm information that accurately narrows down and describes the scope of the failure to relevant personnel as quickly as possible.

[0142] Furthermore, this application can automatically perform in-depth analysis, accurately and comprehensively describing the cause of the fault, the method of fault repair, and initiating an executable script to automatically complete the fault repair through alarms. While transmitting the data to be transmitted to the alarm terminal via alarms, the function of automatically completing the fault repair can be activated under certain conditions, reducing manual intervention. Moreover, through algorithmic analysis of historical alarm data, common false alarm patterns and causes are identified, and alarm noise reduction rules are adjusted in a low-code manner accordingly. Algorithms are reasonably configured and modified for alarm noise reduction and suppression strategies, and similar alarms are aggregated together for dispatch, notification, and processing, greatly improving efficiency, reducing alarm fatigue, and enabling intuitive and efficient closed-loop tracking of troubleshooting these alarm problems.

[0143] For different user roles (such as operations personnel, maintenance personnel, or R&D personnel), this application allows users to subscribe to monitoring rules relevant to their own roles. Subscribing to a unified and standardized monitoring rule template allows for redefinition of alarm levels, notification methods, and callback interfaces, improving alarm response speed. In some examples, the specific implementation steps are as follows:

[0144] Step B1: Set the subscription name and import the alarm policy templates to be subscribed to;

[0145] Step B2: You can re-filter and select alarm monitoring metrics. If a match is found, the data will be transmitted to the alarm terminal. If a match is not found, the data will not be transmitted to the alarm terminal.

[0146] Step B3: The duration of the subscription event can be set. For example, if the configured duration is 100 seconds, the same alarm event will not be considered a successfully matched subscription event the first time it is subscribed. Subsequent times the same subscription event is matched, the difference between the trigger time of the current subscription event and the trigger time of the first subscription will be calculated. If the difference exceeds 100 seconds, the subscription condition will be met, and the alarm notification logic will be executed. If the difference is less than 100 seconds, the subscription event will not be considered successfully matched, and the alarm notification logic will not be executed. This function can be used for alarm escalation. The team leader can configure a subscription with a duration exceeding 1 hour (3600 seconds), with the project leader as the recipient. As the project leader, it is essential to ensure that alarms are followed up. Faults that cannot be resolved after more than 1 hour need to be escalated and reported to the team leader.

[0147] Step B4: As needed, the alarm level, alarm notification method, and callback interface address can be redefined.

[0148] In some embodiments, the indicator data includes indicator values, the device identifier of the indicator source device, and the algorithm code of the target AI visual analysis service. The automated monitoring and maintenance method for AI visual analysis services provided in this application may further include the following steps:

[0149] A visual dependency graph is constructed based on the device identifiers and algorithm codes of each indicator data.

[0150] Based on the visualized dependency graph and various indicator values, a visualized fault diagnosis path is generated.

[0151] In this embodiment, during the process of collecting indicator data, a correspondence between indicator values, device identifiers, and algorithm codes can be generated, so as to facilitate data chaining based on the correspondence and construct a visual dependency graph, thereby solving the problem of data silos.

[0152] In one example, when a user creates a task, the task can be assigned a unique task identifier, associated with device information (i.e., device identifier) ​​and the type of AI vision atomic capability invoked (i.e., algorithm code). This application can combine this task identifier, device identifier, and algorithm code into indicator data, making a set of indicator data an "observation object," and combining it with the strategy logic shown above to determine the health status of the "observation object."

[0153] In some embodiments, please refer to Figure 5 This application can break down functions into multiple sub-modules and thereby achieve automated inspection.

[0154] Persistent time-series databases aggregate disorganized data, massive amounts of metrics, logs, and tracing data. Currently, most industry monitoring platforms present this data flatly, simply categorizing it into metrics, logs, and tracing through menus, and further subdividing metrics into infrastructure, applications, and business subcategories. This application, however, focuses more on the inherent connections and dynamic relationships between data. Users don't need to build complex global models in their minds; they can easily achieve seamless data integration and deep insights through intelligent linkage functions. By cleverly integrating monitoring service quality target metrics and AI vision business metrics, a comprehensive and refined AI vision business metric system is constructed. By enumerating and listing the corresponding result-type metrics and fault categories for core business functions, and by establishing clear monitoring rules for result-type metrics, it is ensured that alarms are triggered whenever a system fault occurs.

[0155] This application utilizes low-code platform tools and data operation and organization to present data characteristics (maximum value, minimum value, year-on-year and month-on-month comparisons, etc.), establishes preliminary monitoring indicators (whether the network is normal, whether the middleware service is normal, whether there are problems with dependent third-party services, etc.), and then integrates all monitoring indicators to construct a visualized dependency graph, forming a relatively good knowledge base. This allows for real-time observation and monitoring of its operational status, ultimately leading to a fault location conclusion. It clearly identifies which specific module of the AI ​​visual analysis service caused the failure and what the cause was. By pinpointing the direct cause of the failure, finding the basis for loss mitigation, and stopping the damage as quickly as possible.

[0156] The method adopted in this application is a knowledge-based reuse strategy, which aims to guide users to first focus on analyzing key data when locating faults, then examine related data in a follow-up manner, and finally form a comprehensive and clear fault diagnosis path by integrating and presenting various types of data.

[0157] First, provide a global cockpit view so that users can see the health status of each business at a glance.

[0158] If the health status of a certain business is shown in red, it indicates an anomaly. Click on details to see which specific core business indicator is abnormal.

[0159] If core business metrics plummet, and at certain times the metrics directly show "0", this is a clear fault. Users can click on the location of the drop to see if the related services (functional interfaces, microservices, middleware, or infrastructure) are healthy. Unhealthy services will be marked in red, allowing users to quickly locate the faulty service.

[0160] Users can click on the red marker to continue drilling down for troubleshooting. They can view logs, trace data, etc., check the relevant logs based on the abnormal indicators, and check the trace data based on the traceID in the logs. Then, by clicking the trace button, they can connect the logs to the trace data and finally find the root cause of the fault.

[0161] The entire process requires only a few clicks from the user, guided by the color-coded health status, allowing for quick fault location. Even maintenance, operations, and development personnel with no prior experience or unfamiliarity with the system can quickly pinpoint the root cause of the problem using this guided approach.

[0162] The automated monitoring and maintenance device for AI visual analysis services provided in the embodiments of this application will be described below. The automated monitoring and maintenance device for AI visual analysis services described below and the automated monitoring and maintenance method for AI visual analysis services described above can be referred to in correspondence.

[0163] In one embodiment, such as Figure 6 As shown, the automated monitoring and maintenance device 700 for AI visual analysis services in this application includes:

[0164] The judgment module 702 is used to determine whether the AI ​​platform generates target alarm data for the target AI visual analysis business in response to the fault diagnosis interface being called or the automatic inspection result being abnormal; wherein, the target AI visual analysis business is the AI ​​visual analysis business that needs to be diagnosed.

[0165] The first fault location module 704 is used to locate the fault based on the subscription information and push logs of the target AI visual analysis service if the AI ​​platform generates the target alarm data, and obtain the fault location result.

[0166] The second fault location module 706 is used to locate the fault based on the task details of the AI ​​platform, the status of the target device, and the status of the target algorithm corresponding to the target AI visual analysis service if the AI ​​platform does not generate the target alarm data, and to obtain the fault location result; wherein, the status of the target device is the status of the device used to run the target AI visual analysis service;

[0167] The push module 708 is used to generate and push fault investigation results based on the fault location results.

[0168] In one embodiment, the first fault location module of this application includes:

[0169] The subscription push judgment unit is used to determine whether the target AI visual analysis service has been subscribed to and whether there is a push anomaly based on the subscription information and the push log, respectively.

[0170] The first location result acquisition unit is used to take the application problem as the fault location result if the target AI visual analysis service has no subscription or has a push anomaly.

[0171] The second location result acquisition unit is used to take the AI ​​middleware push service anomaly as the fault location result if the target AI visual analysis service has been subscribed to and the push log is empty.

[0172] In one embodiment, the task details include a target task status generated by an AI platform for the target AI visual analysis service. The second fault location module of this application includes:

[0173] The third positioning result acquisition unit is used to take the algorithm-side problem as the fault positioning result if the target task status is normal and the target algorithm status is normal, and the business type of the target AI visual analysis business is video business.

[0174] The fourth positioning result acquisition unit is used to, if the target task status is normal and the target algorithm status is normal, then, in the case where the business type is frame-skipping business and there are call logs, take the algorithm-side problem as the fault positioning result, and in the case where the business type is frame-skipping business and there are no call logs, take the algorithm pulling and frame-skipping problem as the fault positioning result.

[0175] The snapshot unit is used to attempt to perform a snapshot operation on the video stream involved in the target AI visual analysis service and obtain the video stream address if the target task status is abnormal, the target algorithm status is push abnormal / the target device status is offline.

[0176] The fifth positioning result acquisition unit is used to take the algorithm's frame pulling problem as the fault positioning result if the capture is successful.

[0177] The sixth positioning result acquisition unit is used to take the video aggregation platform problem / equipment problem as the fault positioning result if the capture fails and the video stream address acquisition fails.

[0178] The seventh positioning result acquisition unit is used to determine whether the frame extraction algorithm can provide normal service based on the failure log if the capture fails but the video stream address is successfully obtained. If the frame extraction algorithm can provide normal service, the network problem / stream problem is used as the fault positioning result. If the frame extraction algorithm is abnormal, the AI ​​platform problem is used as the fault positioning result.

[0179] In some embodiments, the automated monitoring and maintenance device for AI visual analytics services further includes:

[0180] The indicator collection module is used to collect indicator data of the AI ​​visual analysis business according to multiple preset monitoring indicators and a preset collection period; wherein, the multiple monitoring indicators include service quality indicators and business indicators.

[0181] An automatic inspection module is used to perform automated inspections based on the indicator data according to a preset inspection cycle, and obtain the automatic inspection results; the automatic inspection results are either normal or abnormal.

[0182] In one embodiment, the indicator acquisition module of this application includes:

[0183] The multi-probe acquisition unit is used to collect the indicator data of the AI ​​visual analysis service using multiple probe collectors; wherein the multiple probe collectors are configured using the scrape_configs configuration item of the Promethus tool.

[0184] In some embodiments, the automatic inspection module of this application includes:

[0185] The first inspection unit is used to perform automated inspections based on the indicator data;

[0186] The first filtering unit is used to obtain historical alarm data of the target AI visual analysis service if at least one target alarm data is generated during the automated inspection process, and to filter the at least one target alarm data based on the historical alarm data to obtain a filtering result.

[0187] The first inspection result acquisition unit is configured to determine the automatic inspection result as normal if the filtering result is empty, and otherwise determine the automatic inspection result as abnormal.

[0188] In one embodiment, the automatic inspection module of this application includes:

[0189] The second inspection unit is used to perform automated inspections based on the indicator data.

[0190] The second filtering unit is used to perform noise reduction filtering on the at least one target alarm data according to the alarm noise reduction rules pre-configured through the low-code platform if at least one target alarm data is generated during the automated inspection process, and obtain the filtering result.

[0191] The second inspection result acquisition unit is used to determine whether the automatic inspection result is normal if the filtering result is empty, and otherwise determine whether the automatic inspection result is abnormal.

[0192] In some embodiments, the push module of this application includes:

[0193] An alarm push unit is used to push target alarm data from the filter result to the alarm push user according to an alarm push rule defined by the alarm push user if the filter result is not empty; wherein the alarm push rule includes at least one of the following: alarm level, notification method, and callback interface.

[0194] In one embodiment, the indicator data includes indicator values, the device identifier of the indicator source device, and the algorithm code of the target AI visual analysis service. The automated monitoring and maintenance device for the AI ​​visual analysis service of this application further includes:

[0195] The dependency graph construction module is used to construct a visual dependency graph based on the device identifier and algorithm code of each of the aforementioned indicator data.

[0196] The diagnostic path generation module is used to generate a visual fault diagnosis path based on the visualized dependency graph and the values ​​of each indicator.

[0197] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the automated monitoring and maintenance method for AI visual analysis services as described in any embodiment.

[0198] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the automated monitoring and maintenance method for AI visual analysis services as described in any embodiment.

[0199] Indicatively, Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. In one example, the computer device can be a server. (Refer to...) Figure 7 The computer device 900 includes a processing component 902, which further includes one or more processors, and memory resources represented by memory 901 for storing instructions, such as application programs, that can be executed by the processing component 902. The application programs stored in memory 901 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 902 is configured to execute instructions to perform the steps of the automated monitoring and maintenance method for AI visual analysis services described in any of the above embodiments.

[0200] The computer device 900 may also include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate on an operating system stored in memory 901, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0201] Those skilled in the art will understand that the internal structure of the computer device shown in this application is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0202] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0203] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0204] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automated monitoring and maintenance method for AI visual analytics services, characterized in that, include: In response to the fault diagnosis interface being called or the automatic inspection result being abnormal, it is determined whether the AI ​​platform generates target alarm data for the target AI visual analysis business; wherein, the target AI visual analysis business is the AI ​​visual analysis business that needs to be diagnosed. If the AI ​​platform generates the target alarm data, then the fault location is performed based on the subscription information and push logs of the target AI visual analysis service, and the fault location result is obtained. If the AI ​​platform does not generate the target alarm data, then the fault location is performed based on the task details of the AI ​​platform, the target device status, and the target algorithm status corresponding to the target AI visual analysis service, and the fault location result is obtained; wherein, the target device status is the status of the device used to run the target AI visual analysis service, and the task details include the target task status generated by the AI ​​platform for the target AI visual analysis service; Based on the fault location results, generate and push out fault investigation results; The step of locating the fault based on the task details of the AI ​​platform, the status of the target device, and the status of the target algorithm corresponding to the target AI visual analysis service, and obtaining the fault location result, includes: If the target task status is normal and the target algorithm status is normal, then if the business type of the target AI visual analysis service is video service, the algorithm-side problem will be used as the fault location result. If the target task status is normal and the target algorithm status is normal, then if the business type is frame-skipping business and there are call logs, the algorithm-side problem will be used as the fault location result; and if the business type is frame-skipping business and there are no call logs, the algorithm pulling and frame-skipping problem will be used as the fault location result. If the target task status is abnormal, and the target algorithm status is push abnormal or the target device status is offline, then attempt to perform a capture operation on the video stream involved in the target AI visual analysis service and obtain the video stream address. If the capture is successful, the algorithm's frame-pulling problem will be used as the fault location result. If the capture fails and the video stream address fails to be obtained, then the problem with the video aggregation platform or the equipment will be used as the fault location result. If the capture fails but the video stream address is successfully obtained, the failure log is used to determine whether the frame extraction algorithm can provide normal service. If the frame extraction algorithm can provide normal service, the network problem or the stream problem is used as the fault location result. If the frame extraction algorithm is abnormal, the AI ​​platform problem is used as the fault location result.

2. The method according to claim 1, characterized in that, The step of locating the fault based on the subscription information and push logs of the target AI visual analysis service, and obtaining the fault location result, includes: Based on the subscription information and the push logs, determine whether the target AI visual analysis service has been subscribed to and whether there are any push anomalies. If the target AI visual analysis service has no subscription or has abnormal push notifications, the application problem will be used as the fault location result. If the target AI visual analysis service has a subscription and the push log is empty, then the AI ​​middleware push service anomaly will be used as the fault location result.

3. The method according to claim 1, characterized in that, The method further includes: The AI ​​visual analysis service collects indicator data based on multiple preset monitoring indicators and a preset collection period; wherein, the multiple monitoring indicators include service quality indicators and business indicators. According to the preset inspection cycle, an automated inspection is carried out based on the indicator data, and the automatic inspection result is obtained; the automatic inspection result is normal or abnormal.

4. The method according to claim 3, characterized in that, The collected metrics data for the AI ​​visual analysis service include: Multiple probe collectors are used to collect the indicator data of the AI ​​visual analysis service; wherein, the multiple probe collectors are configured using the scrape_configs configuration item of the Promethus tool.

5. The method according to claim 3, characterized in that, The automated inspection based on the indicator data, and the obtaining of the automated inspection results, includes: Automated inspections are performed based on the aforementioned indicator data; If at least one target alarm data is generated during the automated inspection, then the historical alarm data of the target AI visual analysis service is obtained, and the at least one target alarm data is filtered based on the historical alarm data to obtain the filtering result; If the filtering result is empty, the automatic inspection result is normal; otherwise, the automatic inspection result is abnormal.

6. The method according to claim 3, characterized in that, The automated inspection based on the indicator data, and the obtaining of the automated inspection results, includes: Automated inspections are performed based on the aforementioned indicator data; If at least one target alarm data is generated during the automated inspection process, the at least one target alarm data is filtered for noise reduction according to the alarm noise reduction rules pre-configured through the low-code platform to obtain the filtering result; If the filtering result is empty, the automatic inspection result is normal; otherwise, the automatic inspection result is abnormal.

7. The method according to claim 5 or 6, characterized in that, The step of generating and pushing fault diagnosis results based on the fault location results includes: If the filtering result is not empty, the target alarm data in the filtering result is pushed to the alarm push user according to the alarm push rule defined by the alarm push user; wherein, the alarm push rule includes at least one of the following: alarm level, notification method and callback interface.

8. The method according to claim 3, characterized in that, The indicator data includes indicator values, device identifiers of the source devices, and algorithm codes for the target AI visual analysis service; the method further includes: A visual dependency graph is constructed based on the device identifiers and algorithm codes of each of the aforementioned indicator data. Based on the visualized dependency graph and the values ​​of each indicator, a visualized fault diagnosis path is generated.

9. An automated monitoring and maintenance device for AI visual analysis services, characterized in that, include: The judgment module is used to determine whether the AI ​​platform generates target alarm data for the target AI visual analysis business in response to the fault diagnosis interface being called or the automatic inspection result being abnormal; wherein, the target AI visual analysis business is the AI ​​visual analysis business that needs to be diagnosed. The first fault location module is used to locate the fault based on the subscription information and push logs of the target AI visual analysis service if the AI ​​platform generates the target alarm data, and obtain the fault location result. The second fault location module is used to locate the fault based on the task details of the AI ​​platform, the target device status, and the target algorithm status corresponding to the target AI visual analysis service if the AI ​​platform does not generate the target alarm data, and to obtain the fault location result; wherein, the target device status is the status of the device used to run the target AI visual analysis service, and the task details include the target task status generated by the AI ​​platform for the target AI visual analysis service; The push module is used to generate and push fault diagnosis results based on the fault location results. The second fault location module includes: The third positioning result acquisition unit is used to take the algorithm-side problem as the fault positioning result if the target task status is normal and the target algorithm status is normal, and the business type of the target AI visual analysis business is video business. The fourth positioning result acquisition unit is used to, if the target task status is normal and the target algorithm status is normal, then, in the case where the business type is frame-skipping business and there are call logs, take the algorithm-side problem as the fault positioning result, and in the case where the business type is frame-skipping business and there are no call logs, take the algorithm pulling and frame-skipping problem as the fault positioning result. The snapshot unit is used to attempt to perform a snapshot operation on the video stream involved in the target AI visual analysis service and obtain the video stream address if the target task status is abnormal, the target algorithm status is push abnormal, or the target device status is offline. The fifth positioning result acquisition unit is used to take the algorithm's frame pulling problem as the fault positioning result if the capture is successful. The sixth positioning result acquisition unit is used to determine the video aggregation platform problem or equipment problem as the fault positioning result if the capture fails and the video stream address acquisition fails. The seventh positioning result acquisition unit is used to determine whether the frame extraction algorithm can provide normal service based on the failure log if the capture fails but the video stream address is successfully obtained. If the frame extraction algorithm can provide normal service, the network problem or the stream problem is used as the fault positioning result. If the frame extraction algorithm is abnormal, the AI ​​platform problem is used as the fault positioning result.

Images