Video monitoring system intelligent inspection method based on browser automation and visual AI

Abstract

This invention relates to an intelligent inspection method for video surveillance systems based on browser automation and visual AI, comprising the following steps: system initialization configuration; automated login processing; camera list acquisition; video playback test execution; AI visual analysis and judgment; inspection result processing and reporting; system cleanup and log recording. This invention systematically integrates browser automation technology, large language model image recognition, and multi-protocol video adaptation technologies, achieving a good balance in feasibility, practicality, and economy compared to single technical solutions or purely manual operations. This invention can be widely applied to the operation and maintenance management of various video surveillance systems, including smart city public safety monitoring platforms, enterprise security management systems, traffic video surveillance platforms, financial security monitoring systems, and educational institution campus monitoring platforms—scenarios requiring large-scale camera operation and maintenance.

Description

Technical Field

[0001] This invention belongs to the field of computer intelligent algorithm technology, specifically relating to an intelligent inspection method for video surveillance systems based on browser automation and visual AI. Background Technology

[0002] Inspection technology for video surveillance systems is a crucial element in ensuring the continuous and effective operation of security systems. Existing video surveillance system inspection technologies primarily employ the following four implementation schemes: The inspection method is entirely manual. Maintenance personnel manually log into each monitoring platform, click on each camera to check the video status, judge whether the playback is normal by visual observation, record any abnormalities and notify maintenance personnel for handling, and regularly compile inspection reports manually.

[0003] API interface monitoring is a method used by a few monitoring platforms that provide open APIs. It obtains basic information by calling the device status query interface and judges the device's online status based on HTTP status codes and heartbeat detection, but it cannot verify the actual video playback quality.

[0004] Traditional automation scripting methods use web automation tools such as Selenium to write scripts that simulate user login operations, identify specific error messages through page element location and simple image matching, and check the reachability of video stream URLs.

[0005] The professional monitoring software integration method involves purchasing commercial video surveillance operation and maintenance systems and deeply integrating them with monitoring equipment through the manufacturer's dedicated SDK to achieve multi-dimensional monitoring of equipment health. However, this type of solution is usually costly and has limited platform compatibility.

[0006] In summary, existing video surveillance inspection technologies have several problems: In terms of efficiency, manual inspection is slow and incomplete. A complete inspection of a medium-sized monitoring system takes several hours, making it impossible to achieve high-frequency continuous monitoring.

[0007] In terms of technical depth, API methods can only check the device connection status and cannot detect the quality of video content. Traditional automation tools have difficulty identifying visual problems such as image freezing and color abnormalities.

[0008] In terms of compatibility, monitoring platforms from different vendors have significant technical differences, their web interfaces use different technology stacks (React, Vue, etc.), their login mechanisms are diverse (CAPTCHA, dynamic tokens), and their video playback technologies vary (Flash, WebRTC, proprietary protocols), resulting in a lack of a unified adaptation solution.

[0009] Regarding automation barriers, CAPTCHA protection mechanisms severely hinder automated login, traditional OCR has a low recognition rate for complex CAPTCHAs, and anti-scraping strategies limit access frequency.

[0010] In terms of anomaly detection, there is a lack of objective and unified standards, manual judgment is highly subjective, and simple rules cannot identify complex anomaly patterns.

[0011] In terms of system integration, the inspection results are not in a uniform format, making it difficult to interface with the existing operation and maintenance platform, and there is a lack of standardized alarm and fault handling procedures.

[0012] In terms of maintenance costs, the platform interface redesign caused the automated scripts to fail, new devices required manual configuration, and there was a lack of sustainable maintenance mechanisms. Summary of the Invention

[0013] The problem this invention aims to solve is to provide an intelligent inspection method for video surveillance systems based on browser automation and visual AI. This method achieves an end-to-end browser-automated inspection process, forming a standardized inspection workflow; it includes an intelligent CAPTCHA recognition and processing mechanism to improve the accuracy of complex CAPTCHA recognition; a unified adaptation framework for multi-protocol video streams, dynamically selecting the optimal playback scheme based on camera type; it utilizes existing AI image analysis services for multi-dimensional anomaly identification of video screenshots; it features a structured inspection data generation system; a configurable task management engine; it establishes fault tolerance and retry mechanisms; and it facilitates browser environment optimization and configuration.

[0014] To address the aforementioned technical problems, the present invention provides a technical solution: an intelligent inspection method for a video surveillance system based on browser automation and visual AI, comprising the following steps: S1: System initialization configuration, loading the configuration file containing the platform address, login credentials, AI service key, and inspection plan, and initializing the browser instance; S2: Automated login processing. Uses Puppeteer to access the monitoring platform, automatically fills in the username and password, processes the verification code through the image recognition module, and listens for login responses to obtain authentication tokens. S3: Camera list retrieval, calls the platform API to retrieve camera configuration information under the organizational structure, supports filtering cameras to be inspected by quantity limit or priority; S4: Video playback test execution. For each camera, obtain the playback address according to the type, create a page playback container, use the appropriate player to play the video, and take a screenshot and save it after the playback is stable. S5: AI visual analysis and judgment, converts video screenshots to base64 format, calls AI image analysis service, prompts it to identify abnormal situations in the picture, and parses the returned structured analysis results; S6: Inspection result processing and reporting. Summarize the detection results of all cameras and generate structured data containing fields such as device information, detection time, AI analysis results, confidence score, and anomaly description. Report the data to the operation and maintenance management system in batches through a standardized REST API. S7: System cleanup and logging, cleans up temporary screenshot files, closes browser instances, records complete inspection process logs, and supports subsequent troubleshooting and performance analysis.

[0015] Furthermore, S2 includes the following steps: S21: The system starts Puppeteer to create a headless browser instance and navigates to the monitoring platform login page; S22: The image recognition module processes the verification code and automatically fills in the username, password and verification code into the corresponding input boxes; S23: After clicking the login button, wait for the page to redirect. Confirm successful login by checking page elements or URL changes.

[0016] Furthermore, S22 includes the following steps: S21: Detect whether there is a verification code image element on the login page; if so, capture the image. S22: If the verification code is in SVG format, convert it to a standardized PNG image. S23: Input the standardized CAPTCHA image into the large language model visual understanding module to obtain a natural language description of the image content; S24: Parse the characters, numbers, or calculation results contained in the verification code from the natural language description and automatically fill them into the verification code input box.

[0017] Furthermore, S3 includes the following steps: S31: After successful login, the system obtains the camera list using different methods depending on the platform type; S32: For platforms that provide APIs, call the / api / cameras / list interface to obtain structured list data; S33: For pure web interface platforms, the DOM parsing method is used: S34: Find the camera list container on the page and extract the ID, name, location, and other information for each camera element.

[0018] Furthermore, S4 includes the following steps: S41: Identify the video stream protocol type based on the camera configuration information. The protocol type includes national standard GB / T28181, RTSP stream, or manufacturer's proprietary protocol. S42: Based on the protocol type, dynamically select and initialize the corresponding Web player component through the player factory pattern, including playing FLV streams using video.js, creating an iframe to embed the official player, or calling a transcoding service to play RTSP streams; S43: Ensure stable video playback. During the player loading process, configure the browser's autoplay strategy and cross-domain resource sharing strategy, and monitor the player's preparation status, network requests, and error events.

[0019] Furthermore, S5 includes the following steps: S51: After the video plays stably, take a screenshot of the playback area and save it as a PNG file; S52: Send the screenshot to the AI ​​image analysis service interface to identify abnormal screen calls to the AI ​​service interface, parse the returned JSON or text results, and extract the anomaly judgment and confidence score.

[0020] Furthermore, S52 includes the following steps: S521: Convert a screenshot of the video playback area into base64 encoded image data; S522: Construct a request containing the image data and preset analysis prompts, the prompts being used to guide the AI ​​service to analyze from multiple dimensions, including image freeze, error message, image blur, color anomaly, and presence of obstructions; S523: Call the API interface of the AI ​​image analysis service to receive and parse the returned structured or natural language analysis results; S524: Based on the analysis results, determine whether the camera image status is normal, and generate a judgment conclusion that includes the anomaly type and confidence score.

[0021] Furthermore, this invention also provides an intelligent inspection system for video surveillance systems based on browser automation and visual AI, utilizing the aforementioned intelligent inspection method for video surveillance systems based on browser automation and visual AI, including: The configuration management module is used to load and maintain platform connection parameters, login credentials, AI service configurations, and inspection plans. The task scheduling module, connected to the configuration management module, is used to trigger and manage the execution process of inspection tasks according to the plan; The browser control module, connected to the task scheduling module, is used to start and control the headless browser instance and perform page navigation and automated operations. The login authentication module interacts with the browser control module and is responsible for handling form filling and CAPTCHA recognition on the login page; The video playback module is used to adapt video streams of different protocols and control their playback in a browser page; The screenshot capture module is used to capture screenshots of a stable video frame area; The AI ​​service call module is used to send screenshots to external AI services and obtain analysis results; The data reporting module is used to summarize the structured inspection results and report them to the designated operation and maintenance management platform.

[0022] Furthermore, the present invention also provides a computer device, including a memory, a processor, and an algorithm stored in the memory and executable on the processor, wherein the processor implements the above-described data processing method when executing a computer program.

[0023] Furthermore, the present invention also provides a computer-readable storage medium storing a computer algorithm, which, when executed by a processor, performs the above-described data processing.

[0024] 1. This invention systematically integrates browser automation technology, large language model image recognition, and multi-protocol video adaptation technologies, achieving a good balance in feasibility, practicality, and economy compared to single technical solutions or purely manual operations. This invention can be widely applied to the operation and maintenance management of various video surveillance systems, including smart city public safety monitoring platforms, enterprise security management systems, traffic video surveillance platforms, financial security monitoring systems, and educational institution campus monitoring platforms—scenarios requiring large-scale camera operation and maintenance.

[0025] 2. This invention achieves continuous operation and parallel processing capabilities through browser automation technology. In terms of inspection efficiency, it reduces the average inspection time of 3-5 minutes per camera by traditional manual inspection to 1-2 minutes, and reduces the overall inspection time by about 50%, avoiding the waiting and switching time in manual operation.

[0026] 3. In terms of detection accuracy, this invention can identify hidden faults that may be missed by humans, such as image freezing and color abnormalities, through AI visual analysis technology. Compared with methods that rely solely on manual observation or simple rule judgment, the anomaly detection rate is increased by about 30%, while reducing the judgment differences between different people.

[0027] 4. In terms of technical compatibility, compared with the traditional solution that requires each manufacturer to provide specific APIs or SDKs, this method is based on access via a universal browser and can be adapted to most monitoring platforms that support web access. The access time for new platforms is shortened from 1-2 weeks in the traditional solution to 1-3 days.

[0028] 5. In terms of operation and maintenance management, the standardized inspection records automatically generated by this invention contain complete information such as timestamps, camera IDs, AI analysis results, and screenshot evidence, which is more convenient for problem tracing and statistical analysis compared to simple manual recording.

[0029] 6. In terms of cost-effectiveness, the system can replace about 50% of the manual inspection workload, and at the same time, it reduces the duration of monitoring failure by timely fault detection, thus having a good input-output ratio. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of an embodiment of the method of the present invention.

[0031] Figure 2 This is a schematic diagram illustrating the verification code recognition technology principle of an embodiment of the present invention.

[0032] Figure 3 This is an overall structural diagram of an embodiment of the system of the present invention.

[0033] Figure 4 This is a technical schematic diagram of AI visual analysis in an embodiment of the present invention.

[0034] Figure 5 This is a core principle diagram of system integration and data flow in the embodiment of the method of the present invention.

[0035] Figure 6 This is a hierarchical architecture diagram of an embodiment of the system of the present invention. Detailed Implementation

[0036] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The embodiments of the present invention will be further described below with reference to the accompanying drawings: like Figure 1 As shown, the intelligent inspection method for a video surveillance system based on browser automation and visual AI includes the following steps: S1: System initialization configuration, loads the configuration file containing the platform address, login credentials, AI service key, and inspection plan, and initializes the browser instance.

[0038] S2: Automated login processing. Uses Puppeteer to access the monitoring platform, automatically fills in username and password, processes CAPTCHA through image recognition module, and listens for login responses to obtain authentication tokens.

[0039] The core principle of browser automation is to simulate the complete operation chain of a real user through headless browser technology. Puppeteer establishes a communication channel with the browser kernel through the Chrome DevTools Protocol, controlling the browser to load pages, parse the DOM, execute JavaScript, and trigger events—a complete operational process. This mechanism differs from simple HTTP requests; it can fully simulate all user interactions in the browser: from entering username and password to clicking the login button, from navigating to the monitoring page to operating video playback controls. The system ensures that each operation waits for page rendering to complete, network requests to end, and asynchronous operations to be ready through state synchronization mechanisms. Simultaneously, it reduces the risk of being detected by the platform's anti-scraping mechanisms by setting reasonable user agent and viewport size parameters through environment masquerading. This deep browser automation capability enables the system to handle dynamic loading of single-page applications (SPAs) and complex user interaction scenarios common in modern monitoring platforms, laying a technical foundation for subsequent video playback testing.

[0040] Specifically, this embodiment processes the verification code through the image recognition module, including the following steps: Detect whether there is a CAPTCHA image element on the login page, and if so, capture the image.

[0041] If the CAPTCHA is in SVG format, convert it to a standardized PNG image.

[0042] The standardized CAPTCHA image is input into the visual understanding module of the large language model to obtain a natural language description of the image content.

[0043] The system parses the characters, numbers, or calculation results contained in the verification code from the natural language description and automatically fills them into the verification code input box.

[0044] In summary, the technical principle of CAPTCHA recognition adopts a two-stage hybrid approach of "image understanding + semantic parsing". For example... Figure 2As shown, the SVG format CAPTCHA commonly used by monitoring platforms is first converted to the standard PNG image format to solve format compatibility issues. Then, the converted image is input into the visual understanding module of a large language model, allowing the AI ​​to describe the content and meaning of the image. Finally, semantic parsing is performed on the AI's description to extract the actual content of the CAPTCHA (such as numbers, text, or calculation results). Especially for mathematical CAPTCHAs, the system needs to understand and perform arithmetic operations, not just recognize characters. The entire process incorporates a multi-layered fault-tolerance mechanism: automatically refreshing the CAPTCHA for retrying when recognition fails, setting a maximum number of retries to prevent infinite loops, and recording the recognition success rate to optimize subsequent strategies. The advantage of this approach is that it leverages the powerful image understanding capabilities of existing large language models, avoiding the high cost and long cycle of training a dedicated CAPTCHA recognition model, and solving a key obstacle in automated login with relatively simple technical means.

[0045] S3: Camera list retrieval. Calls the platform API to obtain camera configuration information under the organizational structure, and supports filtering cameras to be inspected by quantity limit or priority.

[0046] S4: Video playback test execution. For each camera, obtain the playback address according to the type, create a page playback container, use the adapted player to play the video, and save the screenshot after the playback is stable.

[0047] Specifically, dynamically create or adapt corresponding video player containers based on the video stream protocol type, including: The video stream protocol type is identified based on the camera configuration information. The type includes national standard GB / T28181, RTSP stream, or manufacturer's proprietary protocol.

[0048] Based on the protocol type, the corresponding Web player component is dynamically selected and initialized through the player factory pattern, including playing FLV streams using video.js, creating an iframe to embed the official player, or calling a transcoding service to play RTSP streams.

[0049] During the player loading process, configure the browser's autoplay policy and cross-domain resource sharing policy, and monitor the player's preparation status, network requests, and error events.

[0050] In summary, the core principle of video stream playback adaptation is to establish a unified abstraction layer to achieve a "one-time adaptation, usable in multiple places" playback strategy. For example... Figure 3As shown, the system first identifies the video source type (GB / T28181 national standard, Ezviz proprietary protocol, RTSP stream, etc.) based on the camera configuration information, and then dynamically selects the most suitable player solution: for FLV format GB / T28181 national standard streams, it uses a combination of video.js and flv.js; for Ezviz videos, it uses an iframe to embed the official player; and for RTSP streams, it converts them to a web-compatible format through a transcoding service. During the playback environment preparation phase, the system needs to handle browser autoplay restrictions (bypassing the strategy by setting the muted attribute), resolve cross-domain resource access issues (configuring CORS policies), and ensure the player script loads correctly. During playback, the system continuously monitors the playback status: it detects playback readiness through the readyState attribute of the video element, listens to network request status to track data loading, captures error events to promptly identify playback faults, and calculates playback duration to ensure test sufficiency. This unified adaptation framework enables the system to be compatible with video streams from different manufacturers and using different protocols, providing technical support for large-scale automated testing.

[0051] S5: AI visual analysis and judgment, converts video screenshots to base64 format, calls AI image analysis services such as Tongyi Qianwen, prompts it to identify abnormal situations in the picture, and parses the returned structured analysis results.

[0052] Specifically, screenshots are sent to AI image analysis services to identify anomalies in the image, including: Convert the screenshot of the video playback area into base64 encoded image data.

[0053] Construct a request containing image data and preset analysis prompts. The prompts are used to guide the AI ​​service to analyze multiple dimensions, including image freeze, error message, image blur, color anomaly, and the presence of occlusions.

[0054] Call the API interface of the AI ​​image analysis service to receive and parse the returned structured or natural language analysis results.

[0055] Based on the analysis results, determine whether the camera image is normal and generate a judgment conclusion that includes the anomaly type and confidence score.

[0056] The technical principle of AI visual analysis is to transform the video anomaly detection problem into an image analysis task, utilizing the image understanding capabilities of existing AI services. For example... Figure 3 , Figure 4As shown, the system extracts keyframes from video playback as analysis samples, converts the screenshots into base64 encoded image data, and calls the image analysis interface of a large language model. Through carefully designed prompts, the AI ​​is guided to analyze the image status from multiple dimensions: detecting explicit anomalies such as error messages and connection failures; assessing image quality including blurriness, color anomalies, and brightness issues; identifying content anomalies such as frozen frames, obstructions, and signal loss; and performing time-series analysis through multi-frame comparison to determine dynamic anomalies. The system performs structured parsing of the natural language descriptions returned by the AI, extracting key information and converting it into standardized anomaly classifications and confidence scores. The advantage of this method lies in leveraging the powerful capabilities of existing AI services, avoiding the high cost and complex deployment of building a deep learning model, and implementing professional-grade image analysis functions through API calls, providing intelligent judgment criteria for automated inspection.

[0057] S6: Inspection result processing and reporting. Summarize the detection results of all cameras and generate structured data containing fields such as device information, detection time, AI analysis results, confidence score, and anomaly description. Report the data to the operation and maintenance management system in batches through a standardized REST API.

[0058] The core principle of system integration and data flow is to establish standardized and modular data pipelines to achieve end-to-end automated closed loops. For example... Figure 5 As shown, the data flow follows a strict process: starting with configuration file loading, it proceeds through task initialization, execution process monitoring, result data collection, and standardization processing, ultimately reporting to the operation and maintenance management system. The system adopts a configuration-driven design philosophy, with all runtime parameters stored externally, supporting hot updates without service restarts. It has a built-in, comprehensive state management mechanism that records the execution progress of each task and supports breakpoint resumption after abnormal interruptions. A data normalization module converts inspection results from different platforms and formats into a unified data structure, ensuring consistency in subsequent processing. An asynchronous processing architecture enables non-blocking task execution, improving system throughput and response speed. Error handling employs a tiered strategy: recoverable errors (such as network fluctuations) trigger automatic retries, while unrecoverable errors (such as platform updates) are logged in detail and the administrator is notified. This systematic data flow design ensures the reliability, traceability, and maintainability of the inspection process, providing a solid foundation for large-scale deployment.

[0059] S7: System cleanup and logging, cleans up temporary screenshot files, closes browser instances, records complete inspection process logs, and supports subsequent troubleshooting and performance analysis.

[0060] The entire process supports multiple operating modes, including scheduled automatic triggering, manual immediate execution, and retrying of failed tasks, forming a complete technical closed loop.

[0061] This invention also provides an intelligent inspection system for video surveillance based on browser automation and visual AI, including... The configuration management module is used to load and maintain platform connection parameters, login credentials, AI service configurations, and inspection plans. The task scheduling module, connected to the configuration management module, is used to trigger and manage the execution process of inspection tasks according to the plan; The browser control module, connected to the task scheduling module, is used to start and control the headless browser instance and perform page navigation and automated operations. The login authentication module interacts with the browser control module and is responsible for handling form filling and CAPTCHA recognition on the login page; The video playback module is used to adapt video streams of different protocols and control their playback in a browser page; The screenshot capture module is used to capture screenshots of a stable video frame area; The AI ​​service call module is used to send screenshots to external AI services and obtain analysis results; The data reporting module is used to summarize the structured inspection results and report them to the designated operation and maintenance management platform.

[0062] Its layered architecture system is as follows Figure 6 As shown, the bottom layer is the basic technology support layer, including the browser automation engine and AI service interfaces; the middle layer is the core capability layer, integrating key technology modules such as CAPTCHA recognition, video playback adaptation, and status analysis; the top layer is the application scheduling layer, responsible for task orchestration and process management. The design concept of the entire system is to transform the discrete operations of traditional manual inspection into programmable, repeatable, and scalable automated processes. It is not a simple replacement of manual labor, but rather an optimization and standardization of inspection operations through technological means. This approach retains the flexibility of manual inspection while overcoming its inefficiency and inconsistent standards, forming a quantifiable, traceable, and optimizable technical solution.

[0063] Specifically, the architecture provided in this embodiment is as follows: Figure 3 As shown, the first layer is the system management layer, which includes the configuration management module, task scheduling module, browser control module, and data reporting module.

[0064] The second layer is the browser automation execution layer, which includes a login authentication module, a video playback module, and a screenshot capture module. Preferably, it also includes a page navigation module.

[0065] The third layer is the AI ​​analysis and processing layer, which includes the AI ​​service invocation module. Specifically, the AI ​​service invocation module includes an image preprocessing module, an AI invocation module, a result parsing module, and an anomaly detection module.

[0066] The working process of this embodiment is as follows: Implement environmental preparation First, install the necessary environment on the deployment server: install Node.js version 16.0 or later, install Google Chrome browser for Puppeteer driver, and create a project working directory including a config directory, a logs directory, and a screenshots directory. Install core dependencies such as puppeteer, axios, and moment via npm.

[0067] S1: System Configuration Implementation Create a `config.json` configuration file containing platform access parameters: monitoring platform URL, login username and password, AI service API key and endpoint address, and inspection task scheduling time expression. Create corresponding adapter configuration files for different monitoring platforms, defining the login page element selector, video playback area selector, and camera list retrieval method.

[0068] S2: Implementation of Automated Browser Login The system launches Puppeteer to create a headless browser instance and navigates to the monitoring platform login page. The username and password are automatically filled into the corresponding input boxes. If a CAPTCHA element exists on the page, the CAPTCHA recognition process is executed: the CAPTCHA image is captured, the AI ​​service is called to recognize it, and the recognition result is filled into the CAPTCHA input box. After clicking the login button, wait for the page to redirect. Confirm successful login by checking page elements or URL changes.

[0069] S3: Camera List Acquisition Implementation After successful login, the system retrieves the camera list using different methods depending on the platform type. For platforms that provide an API, the ` / api / cameras / list` interface is called to obtain structured list data. For pure web interface platforms, DOM parsing is used to locate the camera list container on the page and extract information such as the ID, name, and location of each camera element. The system supports both paginated and scroll-based loading scenarios.

[0070] S4: Video Playback Test Implementation For each camera in the list, a playback test is performed sequentially. First, the playback method is determined based on the camera type: standard cameras use the video.js player, while Ezviz cameras are embedded in an iframe. The system creates a temporary playback container on the page, loads the video stream, and waits 10-30 seconds to ensure stable playback. During playback, player status events are monitored to detect buffering, errors, stuttering, and other issues.

[0071] S5: Screenshot and AI Analysis Implementation After the video plays stably, a screenshot of the playback area is taken and saved as a PNG file. The screenshot is converted to base64 encoding, and an AI analysis request is constructed, containing image data and a preset analysis prompt: "Please analyze whether the monitoring screen is normal, and whether there are any error messages, screen freezes, obstructions, or other issues." AI service interfaces such as Tongyi Qianwen are called, and the returned JSON or text results are parsed to extract anomaly detection and confidence scores.

[0072] S6: Result Reporting and Implementation The system aggregates all camera detection results and generates a structured report. The report includes fields such as camera ID, detection timestamp, playback status, AI analysis results, and screenshot file path. The report is then sent to the operations management platform's receiving interface ( / api / inspection / report) via an HTTP POST request. After successful submission, temporary screenshot files are cleared, freeing up browser resources.

[0073] S7: Uses the node-cron library to implement scheduled tasks, automatically executing inspection tasks according to the configured time expression. Detailed logs are recorded in the logs directory during task execution, including start time, end time, number of cameras processed, success / failure statistics, etc. The system provides a manual immediate execution interface, supporting triggering instant inspections via HTTP requests.

[0074] Meanwhile, this embodiment implements a multi-layered fault tolerance mechanism: automatic refresh and retry when CAPTCHA recognition fails, up to 3 times; skipping the current camera and continuing to the next when video playback fails; using backup judgment rules when AI service calls time out or fail; and waiting for reconnection when there is a network anomaly. All abnormal situations are recorded in the error log and do not affect the overall task execution.

[0075] The advantages and positive effects of the invention are: 1. This invention systematically integrates browser automation technology, large language model image recognition, and multi-protocol video adaptation technologies, achieving a good balance in feasibility, practicality, and economy compared to single technical solutions or purely manual operations. This invention can be widely applied to the operation and maintenance management of various video surveillance systems, including smart city public safety monitoring platforms, enterprise security management systems, traffic video surveillance platforms, financial security monitoring systems, and educational institution campus monitoring platforms—scenarios requiring large-scale camera operation and maintenance.

[0076] 2. This invention achieves continuous operation and parallel processing capabilities through browser automation technology. In terms of inspection efficiency, it reduces the average inspection time of 3-5 minutes per camera by traditional manual inspection to 1-2 minutes, and reduces the overall inspection time by about 50%, avoiding the waiting and switching time in manual operation.

[0077] 3. In terms of detection accuracy, this invention can identify hidden faults that may be missed by humans, such as image freezing and color abnormalities, through AI visual analysis technology. Compared with methods that rely solely on manual observation or simple rule judgment, the anomaly detection rate is increased by about 30%, while reducing the judgment differences between different people.

[0078] 4. In terms of technical compatibility, compared with the traditional solution that requires each manufacturer to provide specific APIs or SDKs, this method is based on access via a universal browser and can be adapted to most monitoring platforms that support web access. The access time for new platforms is shortened from 1-2 weeks in the traditional solution to 1-3 days.

[0079] 5. In terms of operation and maintenance management, the standardized inspection records automatically generated by this invention contain complete information such as timestamps, camera IDs, AI analysis results, and screenshot evidence, which is more convenient for problem tracing and statistical analysis compared to simple manual recording.

[0080] 6. In terms of cost-effectiveness, the system can replace about 50% of the manual inspection workload, and at the same time, it reduces the duration of monitoring failure by timely fault detection, thus having a good input-output ratio.

[0081] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An intelligent inspection method for video surveillance systems based on browser automation and visual AI, characterized by: Includes the following steps, S1: System initialization configuration, loading the configuration file containing the platform address, login credentials, AI service key, and inspection plan, and initializing the browser instance; S2: Automated login processing. Uses Puppeteer to access the monitoring platform, automatically fills in the username and password, processes the verification code through the image recognition module, and listens for login responses to obtain authentication tokens. S3: Camera list retrieval, calls the platform API to retrieve camera configuration information under the organizational structure, supports filtering cameras to be inspected by quantity limit or priority; S4: Video playback test execution. For each camera, obtain the playback address according to the type, create a page playback container, use the appropriate player to play the video, and take a screenshot and save it after the playback is stable. S5: AI visual analysis and judgment, converts video screenshots to base64 format, calls AI image analysis service, prompts it to identify abnormal situations in the picture, and parses the returned structured analysis results; S6: Inspection result processing and reporting. Summarize the detection results of all cameras and generate structured data containing fields such as device information, detection time, AI analysis results, confidence score, and anomaly description. Report the data to the operation and maintenance management system in batches through a standardized REST API. S7: System cleanup and logging, cleans up temporary screenshot files, closes browser instances, records complete inspection process logs, and supports subsequent troubleshooting and performance analysis.

2. The intelligent inspection method for a video surveillance system based on browser automation and visual AI as described in claim 1, characterized in that: S2 includes the following steps: S21: The system starts Puppeteer to create a headless browser instance and navigates to the monitoring platform login page; S22: The image recognition module processes the verification code and automatically fills in the username, password and verification code into the corresponding input boxes; S23: After clicking the login button, wait for the page to redirect. Confirm successful login by checking page elements or URL changes.

3. The intelligent inspection method for a video surveillance system based on browser automation and visual AI as described in claim 2, characterized in that: S22 includes the following steps: S21: Detect whether there is a verification code image element on the login page; if so, capture the image. S22: If the verification code is in SVG format, convert it to a standardized PNG image. S23: Input the standardized CAPTCHA image into the large language model visual understanding module to obtain a natural language description of the image content; S24: Parse the characters, numbers, or calculation results contained in the verification code from the natural language description and automatically fill them into the verification code input box.

4. The intelligent inspection method for a video surveillance system based on browser automation and visual AI as described in any one of claims 1 to 3, characterized in that: S3 includes the following steps: S31: After successful login, the system obtains the camera list using different methods depending on the platform type; S32: For platforms that provide APIs, call the / api / cameras / list interface to obtain structured list data; S33: For pure web interface platforms, the DOM parsing method is used: S34: Find the camera list container on the page and extract the ID, name, location, and other information for each camera element.

5. The intelligent inspection method for a video surveillance system based on browser automation and visual AI as described in any one of claims 1 to 3, characterized in that: S4 includes the following steps: S41: Identify the video stream protocol type based on the camera configuration information. The protocol type includes national standard GB / T28181, RTSP stream, or manufacturer's proprietary protocol. S42: Based on the protocol type, dynamically select and initialize the corresponding Web player component through the player factory pattern, including playing FLV streams using video.js, creating an iframe to embed the official player, or calling a transcoding service to play RTSP streams; S43: Ensure stable video playback. During the player loading process, configure the browser's autoplay strategy and cross-domain resource sharing strategy, and monitor the player's preparation status, network requests, and error events.

6. The intelligent inspection method for a video surveillance system based on browser automation and visual AI as described in any one of claims 1 to 3, characterized in that: S5 includes the following steps: S51: After the video plays stably, take a screenshot of the playback area and save it as a PNG file; S52: Send the screenshot to the AI ​​image analysis service interface to identify abnormal screen calls to the AI ​​service interface, parse the returned JSON or text results, and extract the anomaly judgment and confidence score.

7. The intelligent inspection method for a video surveillance system based on browser automation and visual AI as described in claim 6, characterized in that: S52 includes the following steps: S521: Convert a screenshot of the video playback area into base64 encoded image data; S522: Construct a request containing the image data and preset analysis prompts, the prompts being used to guide the AI ​​service to analyze from multiple dimensions, including image freeze, error message, image blur, color anomaly, and presence of obstructions; S523: Call the API interface of the AI ​​image analysis service to receive and parse the returned structured or natural language analysis results; S524: Based on the analysis results, determine whether the camera image status is normal, and generate a judgment conclusion that includes the anomaly type and confidence score.

8. A smart inspection system for video surveillance based on browser automation and visual AI, characterized by: The intelligent inspection method for video surveillance systems based on browser automation and visual AI, as described in any one of claims 1 to 7, include, The configuration management module is used to load and maintain platform connection parameters, login credentials, AI service configurations, and inspection plans. The task scheduling module, connected to the configuration management module, is used to trigger and manage the execution process of inspection tasks according to the plan; The browser control module, connected to the task scheduling module, is used to start and control the headless browser instance and perform page navigation and automated operations. The login authentication module interacts with the browser control module and is responsible for handling form filling and CAPTCHA recognition on the login page; The video playback module is used to adapt video streams of different protocols and control their playback in a browser page; The screenshot capture module is used to capture screenshots of a stable video frame area; The AI ​​service call module is used to send screenshots to external AI services and obtain analysis results; The data reporting module is used to summarize the structured inspection results and report them to the designated operation and maintenance management platform.

9. A computer device comprising a memory, a processor, and an algorithm stored in the memory and executable on the processor, characterized in that: When the processor executes a computer program, it implements the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer algorithm, characterized in that, When the computer algorithm is executed by the processor, it performs the data processing as described in any one of claims 1 to 7.

Images