Large-scale camera intelligent inspection system and method

Abstract

The invention discloses a large-scale camera intelligent inspection system and method, and the system comprises a heterogeneous hardware platform which is provided with a central processing unit and a video acceleration card, and the video acceleration card is integrated with a video processing unit and a neural network processing unit; and the dynamic resource scheduling architecture is constructed on the heterogeneous hardware platform and is provided with an application layer and an infrastructure layer, the application layer comprises an inspection service module, a decoding module and an AI analysis module, and the infrastructure layer receives task state feedback from the decoding module and the AI analysis module by taking Redis as a central event hub. According to the invention, a highly decoupled and horizontally expandable system architecture is constructed to support smooth evolution of the scale of thousands to hundreds of thousands of cameras and ensure long-term stability of the system under high load.

Description

Technical Field

[0001] This invention relates to the field of camera inspection technology, and in particular to a large-scale intelligent camera inspection system and method. Background Technology

[0002] With the rapid development of smart cities and intelligent transportation, the scale of security monitoring networks has exploded. Taking urban rail transit as an example, a single line can deploy thousands of cameras, the total number of cameras in an urban rail transit system can reach tens of thousands, and the total number of cameras in an entire city can reach hundreds of thousands. At the same time, the resolution of cameras is getting higher and higher; 1080P high-definition cameras have been widely deployed, and 4K ultra-high-definition cameras are being deployed at an accelerated pace.

[0003] Traditional operation and maintenance methods rely heavily on manual periodic inspection of the screen, which is extremely inefficient and prone to missing detections. They are completely unable to meet the requirements of modern security systems for real-time performance, accuracy, and comprehensiveness.

[0004] In recent years, some software-based video quality diagnostic tools have emerged, attempting to replace some manual work through automation. These technical solutions mainly fall into two categories, but both have significant drawbacks. They include the following two categories: The first type is a software solution purely based on the central processing unit (CPU) and traditional image processing libraries (such as OpenCV). This type of solution places all the computational load on the CPU, including video stream decoding, image preprocessing, and quality analysis algorithms (such as calculating image sharpness and detecting black screens). Since the CPU is a general-purpose computing unit, it is very inefficient at handling these highly repetitive and computationally intensive tasks, resulting in a severe bottleneck in system throughput. In practice, a mainstream server can only complete image quality analysis of about 35 1080P high-definition video streams per minute, and the CPU utilization rate is consistently above 70%. To cover tens of thousands of cameras, a huge server cluster needs to be deployed, leading to a sharp increase in hardware costs, energy consumption, and data center space usage, making it extremely uneconomical.

[0005] While the second type of approach attempts to accelerate processing using hardware such as graphics processing units (GPUs), it often falls into pitfalls in its choice of technology path. One pitfall is continuing to use traditional, non-intelligent image processing algorithms, simply porting them to GPUs. Although this approach achieves some speedup, its detection accuracy and intelligence level are limited. Its accuracy in identifying complex faults such as "scene changes," "slight blur," and "specific types of occlusion" is far lower than that of deep learning-based methods, resulting in high false positive and false negative rates. Another pitfall is ignoring actual needs and directly using high-end GPUs or dedicated chips designed for training and running ultra-large-scale artificial intelligence models. These chips have powerful computing capabilities but are extremely expensive and consume enormous amounts of power. For the specific task of "video quality diagnosis," the required model complexity is far lower than that of general object recognition. Using such chips is like "using a cannon to kill a mosquito," resulting in a huge waste of computing power and cost, making large-scale application commercially unfeasible.

[0006] In summary, the above solution has the following problems that need to be addressed: 1. How to overcome the performance bottleneck of pure CPU software solutions, achieve high concurrency and real-time quality diagnosis of tens of thousands of camera video streams, and avoid low throughput due to hardware resource limitations.

[0007] 2. In video quality diagnosis tasks, how to avoid the two major pitfalls of insufficient accuracy of traditional image processing algorithms and excessively high hardware costs of large AI models, and find the best balance between accuracy and cost.

[0008] 3. How to build a highly decoupled, horizontally scalable system architecture to support a smooth evolution from thousands to hundreds of thousands of cameras and ensure the long-term stability of the system under high load. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, such as low throughput due to hardware resource limitations, insufficient accuracy of traditional image processing algorithms, and high hardware costs of large AI models.

[0010] In a first aspect, the present invention provides a large-scale intelligent inspection system for cameras, comprising: The heterogeneous hardware platform is equipped with a central processing unit and a video accelerator card. The video accelerator card integrates a video processing unit and a neural network processing unit. The dynamic resource scheduling architecture, built on the heterogeneous hardware platform, consists of an application layer and an infrastructure layer. The application layer includes an inspection service module, a decoding module, and an AI analysis module. The inspection service is used to create tasks, synchronize status, and execute scheduling. The decoding module is used to call the video processing unit to perform hardware-level preprocessing on the video stream, converting the video stream into image data and writing the image data into shared memory. At the same time, it notifies the AI ​​analysis module through a lightweight RPC. The AI ​​analysis module is used to call the neural network processing unit to perform quality diagnosis on the preprocessed image data. The infrastructure layer uses Redis as a central event hub to receive task status feedback from the decoding module and the AI ​​analysis module.

[0011] Optionally, the inspection service module includes a status manager, a backpressure controller, and a scheduler. The status manager is used to continuously synchronize the global status from the infrastructure layer and maintain the real-time status map of the system. The backpressure controller is used to execute the backpressure algorithm based on the global status data and decide whether to allow the allocation of new tasks. The scheduler is used to receive instructions from the backpressure controller and execute specific task allocation actions.

[0012] Optionally, the AI ​​analysis module includes an RPC receiving thread pool and an AI analysis thread pool. The RPC receiving thread pool is used to receive RPC notifications from the decoding module and generate tasks to be analyzed into an internal central task queue. The AI ​​analysis thread pool is used to retrieve tasks from the central task queue and retrieve preprocessed image data from shared memory for quality diagnosis.

[0013] Secondly, the present invention provides a large-scale intelligent inspection method for cameras, characterized by comprising the following steps: The inspection service module creates inspection tasks and assigns them to the decoding module. The decoding module retrieves video stream data from the inspection task and decapsulates it. It then calls the video processing unit to preprocess the video stream data and convert it into image data. The AI ​​analysis module receives image data, analyzes it to obtain diagnostic results, and publishes the diagnostic results to the central event hub. The inspection service module retrieves the diagnostic results from the central event hub and saves the diagnostic results to the business database.

[0014] Optionally, the inspection service module continuously monitors the current effective load in the dynamic resource scheduling architecture through the back pressure controller. If the current effective load does not exceed the maximum concurrency of AI analysis, the task is assigned to the decoding module.

[0015] Optionally, the current effective load is determined based on the number of tasks in decoding and the number of tasks waiting to be analyzed, and the maximum concurrency of AI analysis is determined based on the number of video acceleration cards.

[0016] Optionally, the decoding module and AI analysis module publish the task status to the central event hub, and the inspection service module obtains the task status through the central event hub and updates the task status synchronously.

[0017] Optionally, the decoding module writes the image data into shared memory and notifies the AI ​​analysis module via a lightweight RPC framework, while also passing the task ID and shared memory identifier.

[0018] Optionally, a timeout period can be set for the task during the decoding or analysis process. If the timeout period is exceeded during the decoding or analysis process, the process is deemed to have failed, and the corresponding failed process is restarted.

[0019] Optionally, diagnostic results include device ID, fault type, timestamp, and confidence level.

[0020] The beneficial effects of this invention are as follows: The innovation of this application lies in solving the performance bottleneck problem in high-concurrency real-time quality diagnosis of large-scale camera video streams by combining a heterogeneous hardware platform with a dynamic resource scheduling architecture. Specifically, the heterogeneous hardware platform avoids the inefficient computation of traditional pure CPU solutions and the waste of computing power of high-end GPUs through dedicated hardware acceleration for video processing units and neural network processing units; the dynamic resource scheduling architecture achieves efficient task allocation and status monitoring through modular design, supporting smooth expansion from thousands to hundreds of thousands of cameras. Therefore, this system can significantly improve hardware resource utilization while ensuring diagnostic accuracy and meeting the real-time and stability requirements of large-scale security systems. Attached Figure Description

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Figure 1 These are flowcharts from some of the embodiments; Figure 2 These are system block diagrams from some embodiments; Figure 3 This is another flowchart in some embodiments; Figure 4 These are back pressure control flowcharts for some embodiments. Detailed Implementation

[0023] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0024] This invention provides a large-scale intelligent camera inspection system, comprising: The heterogeneous hardware platform is equipped with a central processing unit and a video accelerator card. The video accelerator card integrates a video processing unit and a neural network processing unit. The dynamic resource scheduling architecture, built on the heterogeneous hardware platform, consists of an application layer and an infrastructure layer. The application layer includes an inspection service module, a decoding module, and an AI analysis module. The inspection service is used to create tasks, synchronize status, and execute scheduling. The decoding module is used to call the video processing unit to perform hardware-level preprocessing on the video stream, converting the video stream into image data and writing the image data into shared memory. At the same time, it notifies the AI ​​analysis module through a lightweight RPC. The AI ​​analysis module is used to call the neural network processing unit to perform quality diagnosis on the preprocessed image data. The infrastructure layer uses Redis as a central event hub to receive task status feedback from the decoding module and the AI ​​analysis module.

[0025] In practical applications, a heterogeneous hardware platform can be understood as a hardware architecture composed of various different computing units, which optimizes performance by allocating different types of tasks to suitable computing units. For example, the central processing unit can be a general-purpose CPU, responsible for task scheduling and system management; the video accelerator card can be an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), where the video processing unit (VPU) can be dedicated to hardware-level high-speed video decoding, color space conversion, image scaling, and JPEG encoding, and the neural network processing unit (NPU) is optimized for efficiently executing lightweight neural network models, providing just the right amount of AI computing power. The main purpose of this hardware design is to improve task processing efficiency through dedicated hardware acceleration while reducing overall energy consumption.

[0026] Furthermore, the dynamic resource scheduling architecture is a layered system architecture that achieves efficient task allocation and status monitoring through the collaborative work of the application layer and the infrastructure layer. Specifically, the inspection service module can create and allocate tasks through a distributed message queue, the decoding module can use multi-threading technology to process multiple video streams in parallel, and the AI ​​analysis module can use batch processing to analyze preprocessed image data.

[0027] The innovation of this application lies in solving the performance bottleneck problem in high-concurrency real-time quality diagnosis of large-scale camera video streams by combining a heterogeneous hardware platform with a dynamic resource scheduling architecture. Specifically, the heterogeneous hardware platform avoids the inefficient computation of traditional pure CPU solutions and the waste of computing power of high-end GPUs through dedicated hardware acceleration for video processing units and neural network processing units; the dynamic resource scheduling architecture achieves efficient task allocation and status monitoring through modular design, supporting smooth expansion from thousands to hundreds of thousands of cameras. As a result, the system can significantly improve hardware resource utilization while ensuring diagnostic accuracy and meeting the real-time and stability requirements of large-scale security systems.

[0028] The working principle of this application embodiment is as follows: A large-scale intelligent camera inspection system solves the problem of high-concurrency real-time quality diagnosis of large-scale camera video streams through the collaborative design of a heterogeneous hardware platform and a dynamic resource scheduling architecture. The heterogeneous hardware platform consists of a central processing unit (CPU) and a video accelerator card, with the video accelerator card integrating a video processing unit and a neural network processing unit. The CPU is mainly responsible for general task scheduling, the video processing unit is dedicated to hardware-level preprocessing of the video stream, and the neural network processing unit performs quality diagnosis tasks on the preprocessed image data. This avoids the inefficiency of pure CPU solutions when handling computationally intensive tasks, while also avoiding the cost waste caused by excessive computing power of high-end GPUs. The dynamic resource scheduling architecture is built on the heterogeneous hardware platform and is divided into an application layer and an infrastructure layer. The application layer includes an inspection service module, a decoding module, and an AI analysis module. The inspection service module creates tasks, synchronizes status, and executes scheduling. The decoding module calls the video processing unit to perform hardware-level preprocessing on the video stream, converting it into image data and writing it to shared memory while simultaneously notifying the AI ​​analysis module via lightweight RPC. The AI ​​analysis module calls the neural network processing unit to perform quality diagnosis on the preprocessed image data. Furthermore, the infrastructure layer uses Redis as a central event hub to centrally receive task status feedback from the decoding and AI analysis modules. Specifically, the inspection service module monitors task progress in real time based on feedback from the central event hub, while the decoding and AI analysis modules achieve parallel processing through hardware acceleration. The preprocessed image data is directly transmitted to the AI ​​analysis stage, ensuring efficient task allocation and status monitoring. In addition, Redis, as the central event hub, provides a low-latency status information synchronization mechanism, supporting the system's elastic scalability. Thus, the system achieves high-concurrency processing of tens of thousands of video streams and can smoothly scale to a larger scale according to actual needs. The overall technical solution breaks through the throughput limitations of traditional solutions by using dedicated hardware acceleration on a heterogeneous hardware platform. The modular design of the dynamic resource scheduling architecture improves the efficiency of task allocation and status monitoring, jointly solving the problem of low throughput caused by hardware resource limitations. At the same time, it balances detection accuracy and hardware cost, providing technical support for smooth expansion from thousands to hundreds of thousands of cameras.

[0029] like Figure 2 As shown, the application layer further includes three core business components: an inspection service module, a decoding module, and an AI analysis module. They interact through the principle of separating control flow, data flow, and state flow. The control flow represents the inspection service module's handling of task scheduling control, the data flow represents the decoding module and AI analysis module's handling of video stream data, and the state flow represents the application layer's interaction with the infrastructure layer to provide feedback on task status.

[0030] In some embodiments, the inspection service module includes a state manager, a backpressure controller, and a scheduler. The state manager is used to continuously synchronize the global state from the infrastructure layer and maintain the real-time status map of the system. The backpressure controller is used to execute the backpressure algorithm based on the global state data and decide whether to allow the allocation of new tasks. The scheduler is used to receive instructions from the backpressure controller and execute specific task allocation actions.

[0031] Specifically, a state manager is a component that can track and update the task execution phase in real time. It can be implemented using an event-driven state update mechanism or a periodic polling method to collect task status information. Its purpose is to ensure the system accurately grasps task progress and avoids resource misjudgments due to lagging status information. A backpressure controller is a component used to sense system load pressure. It can be implemented by monitoring real-time indicators such as the number of tasks in decoding and the number of tasks waiting for analysis, or by using a sliding window statistical method. Its purpose is to provide timely and reliable load information for scheduling decisions. A scheduler is a component that dynamically adjusts the task distribution strategy based on load status. It can be implemented using a threshold-based task distribution control mechanism or an adaptive task queue management method. Its purpose is to prevent resource overload and improve system stability in high-concurrency scenarios.

[0032] In detail, the above solution addresses the challenges posed by dynamic load changes during large-scale inspections by constructing a closed-loop control mechanism. The state manager continuously synchronizes the global state from the infrastructure layer, maintaining a real-time system status map, enabling the system to accurately track the execution stage of each task. The backpressure controller executes a backpressure algorithm based on the global state data, deciding whether to allow the allocation of new tasks. The scheduler receives instructions from the backpressure controller and executes specific task allocation actions. This allocation strategy, which dynamically adjusts based on real-time load, fundamentally avoids resource overload caused by blind task distribution and forms a close linkage with the progress data of the state manager, significantly improving the system's robustness and resource utilization efficiency in high-concurrency scenarios. Furthermore, this solution, combined with a heterogeneous hardware platform and dynamic resource scheduling architecture, solves the load fluctuation problem caused by the expansion of camera scale through real-time perception and proactive control capabilities, effectively supporting high-concurrency stable inspections of tens of thousands of cameras. The state manager, backpressure controller, and scheduler adopt a policy-perception-execution separation design.

[0033] In some embodiments, the AI ​​analysis module includes an RPC receiving thread pool and an AI analysis thread pool. The RPC receiving thread pool is used to receive RPC notifications from the decoding module and generate tasks to be analyzed into an internal central task queue. The AI ​​analysis thread pool is used to retrieve tasks from the central task queue and retrieve preprocessed image data from shared memory for quality diagnosis.

[0034] Among them, the RPC receive thread pool refers to a multi-threaded management mechanism specifically designed for handling remote procedure call (RPC) receiving tasks. It can be implemented using a fixed-size thread pool or a dynamically adjusted thread pool, aiming to improve task receiving efficiency through concurrent processing. The AI ​​analysis thread pool is a multi-threaded management mechanism specifically designed for executing neural network analysis tasks. It can optimize task allocation by setting priority queues or load balancing strategies, aiming to improve the utilization of neural network processing units and task processing speed.

[0035] Specifically, this solution improves task scheduling efficiency in high-concurrency scenarios by separating the internal task processing flow of the AI ​​analysis module and constructing a dual-thread pool architecture. The RPC receiving thread pool is dedicated to receiving RPC notifications from the decoding module, avoiding blocking of task reception by analysis operations and ensuring the rapid generation of tasks to be analyzed into the central task queue, thus maintaining the continuity and timeliness of task input. The AI ​​analysis thread pool actively retrieves tasks from the central task queue and obtains preprocessed image data from shared memory for quality diagnosis, focusing on calling the neural network processing unit to perform quality diagnosis. This task acquisition mechanism based on the receiving thread pool achieves dynamic decoupling of task distribution, allowing the receiving and analysis stages to operate in parallel, effectively reducing resource waiting time and balancing hardware load. Overall, this design optimizes the utilization of the neural network processing unit through task queue buffering and elastic thread pool scheduling, enhancing the system's stable processing capability under a scale of tens of thousands of cameras. Furthermore, this solution works organically with the neural network processing unit in heterogeneous hardware platforms, significantly improving the real-time performance and throughput of large-scale video stream processing through reasonable task allocation and resource scheduling, solving the problems of receiving blockage, resource contention, and task backlog inherent in traditional solutions.

[0036] This invention also provides a large-scale intelligent inspection method for cameras, comprising the following steps: S1. The inspection service module creates inspection tasks and assigns them to the decoding module. S2. The decoding module pulls the video stream data from the inspection task and decapsulates it. It then calls the video processing unit to preprocess the video stream data and convert it into image data. The S3 AI analysis module receives image data, analyzes it to obtain diagnostic results, and publishes the diagnostic results to the central event hub. The inspection service module obtains the diagnostic results from the central event hub and saves the diagnostic results to the business database.

[0037] The core innovation of this application lies in effectively solving the key problem of high-concurrency real-time diagnosis of large-scale video streams by combining the task scheduling mechanism of the inspection service module, the hardware-accelerated preprocessing of the decoding module, and the intelligent diagnosis of the AI ​​analysis module in a collaborative process. Specifically, this method avoids the performance bottleneck of pure CPU processing, avoids the shortcomings of insufficient accuracy and excessive cost of high-end hardware in traditional algorithms, and ensures system scalability through modular design. For example, the inspection service module realizes centralized scheduling and orderly allocation of tasks, providing a foundation for high-concurrency processing; the decoding module uses the video processing unit to improve decoding efficiency, overcomes the CPU performance limitations, and achieves efficient conversion of video streams; the AI ​​analysis module ensures diagnostic accuracy through the neural network processing unit, and the central event hub uniformly manages the state flow, ensuring reliable transmission and storage of results. These features work together to build an efficient, stable, and scalable diagnostic process, effectively addressing the real-time, accuracy, and scalability requirements in large-scale camera scenarios.

[0038] In practical applications, a heterogeneous hardware platform can be understood as a hardware architecture composed of various different computing units, which optimizes performance by allocating different types of tasks to suitable computing units. For example, the central processing unit can be a general-purpose CPU, responsible for task scheduling and system management; the video accelerator card can be an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), where the video processing unit is dedicated to hardware-level high-speed video decoding, color space conversion, image scaling, and JPEG encoding, and the neural network processing unit is optimized for efficiently executing lightweight neural network models, providing just the right amount of AI computing power. The main purpose of this hardware design is to improve task processing efficiency through dedicated hardware acceleration while reducing overall energy consumption.

[0039] In summary, this application addresses the performance bottleneck in high-concurrency real-time quality diagnosis of large-scale camera video streams by combining a heterogeneous hardware platform with a dynamic resource scheduling architecture. Specifically, the heterogeneous hardware platform avoids the inefficient computation of traditional pure CPU solutions and the wasted computing power of high-end GPUs through dedicated hardware acceleration for video processing units and neural network processing units. The dynamic resource scheduling architecture achieves efficient task allocation and status monitoring through modular design, supporting smooth expansion from thousands to hundreds of thousands of cameras. Therefore, this system can significantly improve hardware resource utilization while ensuring diagnostic accuracy and meeting the real-time and stability requirements of large-scale security systems.

[0040] In some embodiments, the inspection service module continuously monitors the current effective load in the dynamic resource scheduling architecture through the back pressure controller. If the current effective load does not exceed the maximum concurrency of AI analysis, the task is assigned to the decoding module.

[0041] Specifically, a backpressure controller is a mechanism for real-time monitoring and regulation of system load. It can be implemented using methods such as queue length detection, task completion time statistics, or resource utilization assessment. In practical applications, the design purpose of a backpressure controller is to ensure that the system can dynamically adjust its task allocation strategy based on the current actual load, thereby avoiding task backlog or resource exhaustion due to overload. The current effective load can be understood as the total number of tasks the system is processing at a given moment. It can be quantified by statistically analyzing the number of tasks in decoding and the number of tasks waiting for analysis, aiming to accurately reflect the system's real-time processing pressure. The maximum concurrency of AI analysis refers to the maximum number of quality diagnostic tasks the system can process simultaneously under hardware resource constraints. It can be set based on the number of video accelerator cards to ensure that task allocation is always within the hardware's capabilities.

[0042] In detail, this solution introduces a backpressure controller to build a dynamic task allocation mechanism, enabling the inspection service module to perceive the system load status in real time and respond accordingly. The backpressure controller continuously collects key indicators in the dynamic resource scheduling architecture, including the operating status of the decoding module and the AI ​​analysis module, and compares this data with the preset maximum concurrency of AI analysis. When the current effective load is detected to be below this threshold, a task allocation operation is triggered, distributing new inspection tasks to the decoding module. This design not only ensures that the task flow is always controlled within the system's carrying capacity, but also fully utilizes the computing power of heterogeneous hardware platforms, achieving efficient resource utilization. When the current effective load is detected to be above this threshold, the task allocation operation is paused, preventing data backlog from the source and ensuring system stability. Furthermore, by combining with the dynamic resource scheduling architecture, this mechanism can flexibly adapt to the inspection needs of cameras of different scales, thereby ensuring the long-term stability and reliability of the system during large-scale inspections.

[0043] In some embodiments, the current payload is determined based on the number of tasks in decoding and the number of tasks waiting to be analyzed, and the maximum concurrency of AI analysis is determined based on the number of video acceleration cards.

[0044] Specifically, the current effective load refers to the total number of tasks the system is processing at a given moment. This can be achieved by summing the number of tasks in decoding and the number of tasks waiting to be analyzed, aiming to accurately reflect the actual workload of the heterogeneous hardware platform. The number of tasks in decoding refers to the number of tasks currently calling the video processing unit for preprocessing, while the number of tasks waiting to be analyzed refers to the number of tasks waiting to be called by the neural network processing unit for quality diagnosis after decoding. By separately counting the number of tasks in these two stages, the number of tasks about to enter AI analysis can be accurately reflected. The maximum concurrency of AI analysis can be understood as the maximum number of quality diagnosis tasks that the system can process simultaneously under hardware resource constraints. This can be determined by the number of video accelerator cards, aiming to ensure that task allocation neither exceeds the hardware's carrying capacity nor wastes available resources.

[0045] In detail, the above technical solution directly correlates the current effective load with the number of tasks in decoding and the number of tasks waiting to be analyzed, enabling the backpressure controller to more accurately perceive dynamic load changes in the system. For example, when the number of tasks in decoding is significantly higher than the number of tasks waiting to be analyzed, it indicates that the video processing unit is under significant pressure, while the neural network processing unit is relatively idle. In this case, the backpressure controller can pause the allocation of new tasks to the decoding module to avoid overloading the video processing unit. Conversely, if the number of tasks waiting to be analyzed is high, it indicates that the neural network processing unit has become a bottleneck. In this case, task allocation can be appropriately reduced to balance the load. In addition, the maximum concurrency of AI analysis is determined by the number of video acceleration cards, ensuring that the system can flexibly adjust the concurrency limit according to the hardware configuration. For example, when the number of video acceleration cards is increased, the system can automatically increase the concurrency threshold to fully utilize the increased computing power; adapting to inspection needs of different scales, thereby always keeping task allocation within the optimal range.

[0046] Through the above technical solutions, the system can not only dynamically match hardware resources with task requirements, but also maintain stability and efficiency under high load, solve the problem of task allocation imbalance caused by parameter ambiguity, and optimize the scheduling efficiency of large-scale video inspection.

[0047] In some embodiments, the decoding module and the AI ​​analysis module publish the task status to the central event hub, and the inspection service module obtains the task status through the central event hub and updates the task status synchronously.

[0048] Specifically, task status refers to the execution status at each stage of video stream processing, which can include intermediate states such as decoding, waiting for analysis, and analysis. In practical applications, task status can be implemented using structured data formats, such as JSON or Protobuf, facilitating reliable transmission in asynchronous high-concurrency environments. The central event hub acts as a status relay station, aiming to avoid the strong coupling problems caused by direct communication between modules, while ensuring the real-time consistency of status information.

[0049] In detail, this solution addresses the lack of dynamic system load awareness by constructing a closed-loop monitoring mechanism for task status. The decoding and AI analysis modules publish intermediate states during task execution to the central event hub in real time, enabling centralized recording and management of all task states. The inspection service module proactively retrieves status information from the central event hub, rather than relying on module callbacks. This decoupled approach allows for flexible, on-demand monitoring of the global load. Simultaneously, by synchronously updating task status, the internal state of the inspection service module is ensured to be strictly consistent with the actual execution progress, providing the backpressure controller with accurate current effective load data. This mechanism is particularly crucial in the aforementioned large-scale intelligent camera inspection method. When the system needs to handle hundreds of thousands of cameras, it can dynamically adjust resource scheduling decisions based on the latest system status, effectively preventing task backlog and resource overload, thereby maintaining long-term stable system operation.

[0050] In some embodiments, the decoding module writes image data into shared memory and notifies the AI ​​analysis module through a lightweight RPC framework, while also passing the task ID and shared memory identifier.

[0051] Specifically, shared memory refers to a mechanism that allows multiple processes to share the same physical memory region. It can be implemented using shared memory segments or memory-mapped files provided by the operating system. Its purpose is to avoid redundant data copying operations between processes in traditional methods, thereby significantly reducing CPU load and latency. A lightweight RPC framework can be understood as a remote procedure call protocol. It can achieve a low-latency notification mechanism by simplifying the communication protocol stack and reducing serialization overhead. Its purpose is to replace heavy network communication, ensuring immediate triggering and efficient response of notification commands. A task ID is an identifier used to uniquely identify the context of an inspection task. It can be implemented using a globally unique identifier generation algorithm, enabling the AI ​​analysis module to accurately associate task contexts. A shared memory identifier is a pointer to the physical location of data in shared memory. It can be implemented using memory address offsets or handles, aiming to clearly define the specific storage location of the data and ensure the accuracy of data access.

[0052] Specifically, the above solution optimizes the data transfer path in large-scale video stream processing by combining shared memory with a lightweight RPC framework. The decoding module writes image data directly to shared memory, avoiding redundant data copying between processes in traditional methods, achieving zero-copy transmission, and thus significantly reducing CPU resource consumption and data transfer latency. The lightweight RPC framework is responsible for notifying the AI ​​analysis module, using a low-overhead remote call mechanism to replace traditional heavy network communication or message queues, ensuring that notification commands are triggered quickly and receive efficient responses. Meanwhile, the transmission of task IDs and shared memory identifiers further enhances system reliability. The task ID allows the AI ​​analysis module to accurately associate the context information of the inspection task, while the shared memory identifier clarifies the specific location of the data in shared memory. The combination of these two effectively prevents task corruption or data loss in high-concurrency scenarios. Furthermore, this solution, combined with a heterogeneous hardware platform and dynamic resource scheduling architecture, fully utilizes the characteristics of hardware resources, improves system scalability and stability, and provides a solid foundation for supporting high-concurrency processing of hundreds of thousands of cameras.

[0053] In some embodiments, a timeout period is set for the task during the decoding or analysis process. If the timeout period is exceeded during the decoding or analysis process, the processing is determined to have failed, and the corresponding failed process is restarted.

[0054] Specifically, timeout refers to the time threshold set for task execution, which can be implemented using a fixed time window, a dynamically adjusted time window, or an adaptive time window based on task complexity. In practical applications, the purpose of introducing timeout is to prevent tasks from being suspended indefinitely due to hardware failures, network fluctuations, or resource contention, thereby ensuring the effective utilization of system resources and the continuity of the inspection process.

[0055] In detail, this solution effectively addresses task execution anomalies by setting explicit time limits for the decoding or analysis process. When a task fails to complete within the specified time, the system automatically determines it as a processing failure and triggers the corresponding restart mechanism. This mechanism not only enables rapid recovery of task execution in the event of a momentary failure in hardware acceleration units (such as video processing units or neural network processing units), but also prevents task backlog caused by resource contention or overload. Furthermore, the timeout mechanism design for the two critical steps of decoding and analysis fully considers the collaborative operation characteristics of heterogeneous hardware platforms, ensuring a balance between resource utilization and task success rate in high-concurrency scenarios. Through the above technical solutions, the system can achieve higher stability and real-time performance in large-scale camera inspection scenarios, meeting the needs of concurrent processing of tens of thousands of video streams, while significantly reducing the cost of manual intervention.

[0056] In some embodiments, the diagnostic results include device ID, fault type, timestamp, and confidence level.

[0057] Specifically, the device ID refers to the coded information used to uniquely identify each camera. It can be implemented using a MAC address, serial number, or custom number, aiming to ensure that diagnostic results are accurately associated with a specific camera and avoid confusion caused by missing device identifiers in large-scale systems. The fault type refers to the specific classification of video quality anomalies, such as black screen, blur, or obstruction. This can be implemented through preset fault codes or a tagging system, aiming to support rapid identification of fault categories and trigger corresponding maintenance actions in subsequent automated processing. The timestamp is a time stamp recording the specific moment the diagnosis occurred, which can be implemented using UTC or local time formats, aiming to support time-series-based fault trend analysis and timeliness verification. Confidence level is a quantitative assessment of the reliability of the diagnostic results, usually expressed as a percentage. It can be implemented through probabilistic statistical models or threshold settings, aiming to enable the system to dynamically adjust decision-making strategies based on the confidence level, thereby improving the overall reliability of the diagnosis and maintenance efficiency.

[0058] In detail, the above solution addresses the issues of incomplete diagnostic information and insufficient standardization by defining the core elements of the diagnostic results in a structured manner. The device ID, as a key identifier, ensures that the business database can accurately trace the source of the fault; the clear classification of fault types facilitates the system's rapid identification of the problem's nature and the implementation of corresponding measures; timestamps provide a time-dimensional reference for fault analysis, supporting historical data backtracking and trend prediction; and the introduction of confidence levels provides a quantitative standard for the reliability of diagnostic results, helping to filter out low-confidence results and reduce false alarms. These elements together constitute a complete diagnostic result structure, effectively avoiding data chaos and system maintenance difficulties in large-scale camera deployment scenarios, significantly improving system maintainability and diagnostic efficiency.

[0059] Building upon this foundation, the structured definition of diagnostic results, combined with the intelligent diagnostic capabilities of the AI ​​analysis module, forms a complete video quality diagnostic solution. By standardizing and storing the diagnostic results in the business database, not only is subsequent fault analysis and operational decisions supported, but a reliable data foundation is also provided for the long-term stable operation of the system.

[0060] Practical application scenario 1: In a city rail transit security project, daily inspections of over 20,000 cameras are required. A server equipped with four (N=4) video acceleration cards is deployed. The number of video acceleration cards is determined by the scale of the inspection project; configurations of eight or twelve cards can achieve system scalability. N represents the number of video acceleration cards, specifically dedicated video acceleration cards (e.g., NETINT Quadra) for the Video Processing Unit (VPU) and Neural Processing Unit (NPU). This configuration is chosen as a typical embodiment to illustrate that the invention can operate efficiently in a common, medium-performance hardware environment.

[0061] 1. System Initialization After the system powers on, the inspection service starts. Based on the number of video acceleration cards in the system, N=4, it creates a resource pool according to preset resource pooling rules.

[0062] Decoding Module (Decoding Thread Pool): Starts a decoding thread, which internally creates a thread pool containing 100 independent decoding threads based on the creation parameters K = N * M = 4 * 25 = 100. All threads are initially in the "idle" state. Here, K is the maximum decoding concurrency, and M is the number of decoding threads per video accelerator card.

[0063] AI Analysis Module: Starts an AI analysis module process, which internally creates a thread pool containing 48 analysis threads based on the parameters L = N * P = 4 * 12 = 48. All threads are initially in the "idle" state. L is the maximum concurrency of AI analysis, and P is the number of analysis threads per video accelerator card.

[0064] 2. Task scheduling and execution Task Creation: The inspection service module obtains a resource list of 20,000 cameras from the CCTV video surveillance platform based on the GB / T 28181 protocol (GB / T 28181 protocol is officially called "Technical Requirements for Information Transmission, Exchange and Control of Security Video Surveillance Network System"), and concatenates them into RTSP addresses (Real-Time Streaming Protocol). It creates inspection tasks for these cameras, and the initial status of all tasks is "Ready".

[0065] Back pressure control and task allocation: The back pressure controller inside the inspection service continuously monitors the overall status. When the system load allows, the scheduler assigns "ready" tasks to idle decoding threads and updates the task status to "decoding".

[0066] The decoding module first preprocesses the video stream: each decoding thread is responsible for one video stream. It performs the following core preprocessing steps: The video stream is retrieved and decapsulated using the RTSP protocol. The dedicated video accelerator card's VPU is used for hardware decoding, outputting raw YUV data, i.e., the video stream's luminance and chrominance signal data; Finally, the processed image data is compressed into JPEG format using a VPU. This process compresses the original Gbps-level video data into MB-level image data, greatly reducing the computational burden and transmission pressure on subsequent AI analysis.

[0067] 3. Efficient data transmission and AI analysis: After the decoding thread completes preprocessing, it writes the JPG data to shared memory and then directly notifies the AI ​​analysis module via lightweight RPC, transmitting the task ID and shared memory identifier. This separates control signals from the large data volume, ensuring minimal latency in data transmission.

[0068] The decoding thread simultaneously publishes a task status update event (waiting for analysis) to Redis. After the status manager of the inspection service listens for this event, it updates the task status to "waiting for analysis".

[0069] The AI ​​analysis module's RPC receiving thread generates tasks to be analyzed and adds them to an internal central task queue. Idle AI analysis threads compete to retrieve tasks from the queue and first publish a task status update event (under analysis) to Redis. Upon receiving this event, the status manager of the inspection service updates the task status to "under analysis".

[0070] The AI ​​analysis thread reads JPG data from shared memory, calls the NPU of a dedicated accelerator card, loads and executes a lightweight AI model (such as an optimized YOLOv5s), and diagnoses the image quality.

[0071] 4. Closed-loop system and resource recycling: After the AI ​​analysis thread completes its analysis, it publishes a task status update (complete) event to Redis, which includes the diagnostic results (such as fault type, confidence level, etc.).

[0072] Upon receiving this event, the inspection service's status manager updates the task status to "Completed," saves the analysis results, and sets the corresponding decoding thread status to "Idle." After receiving the analysis results, the inspection service can choose to persistently store the structured diagnostic results (such as device ID, fault type, timestamp, and confidence level) in a business database (such as MySQL) for upper-layer applications to perform historical queries, generate statistical reports, and visualize the data. This persistence process is executed asynchronously with the core real-time scheduling process and does not affect system throughput.

[0073] Backpressure Release: This process decrements the "Current Effective Load" counter by 1. If the system was previously under backpressure, it immediately and automatically resumes the allocation of new tasks, forming an efficient closed-loop scheduling. The inspection service ensures that the global "Current Effective Load" count never exceeds the maximum concurrency of AI analysis, which is the cornerstone of the system's high stability. Current Effective Load = Number of tasks in decoding + Number of tasks waiting for analysis. The backpressure controller ensures that the current effective load is ≤ the maximum concurrency of AI analysis.

[0074] Furthermore, this invention uses an independent monitoring thread to set a timeout for each state (e.g., a 5-minute timeout for the "Analyzing" state). Timeout tasks are automatically marked as "failed," and their associated decoding and analysis tasks are forcibly restarted, ensuring that individual failures do not affect the overall system.

[0075] Practical application scenario 2: The system described in this invention has excellent horizontal scalability. In scenarios requiring the handling of a larger number of cameras (e.g., more than 50,000) or a shorter inspection cycle, performance can be linearly improved by increasing hardware resources.

[0076] Scale-Up (Enhanced Standalone): Up to 8 video accelerator cards can be added to the same server (N=8). During system initialization, 8 * 25 = 200 decoding threads and 8 * 12 = 96 AI analysis threads will be automatically created, theoretically doubling the throughput compared to N=4.

[0077] Scale-Out (Multi-Machine Cluster): A cluster can be formed by adding server nodes. New nodes only need to install this system and connect to the same Redis central event hub to integrate into the system as new resource providers. They are scheduled by a unified inspection service, achieving near-infinite linear scalability.

[0078] Practical application scenario 3 (effect verification): To verify the actual performance of this system, conservative parameters were used for calculation based on the configuration of Example 1 (N=4). With the AI ​​analysis single-channel time set at 30 seconds and video preprocessing time at several seconds, the AI ​​analysis module (48 concurrent channels) ensured that the overall stable throughput of the system was no less than 96 channels / minute.

[0079] Based on this calculation, the theoretical total number of inspections per day (24 hours) for a single server is no less than: 96 channels / minute × 60 minutes × 24 hours = 138,240 channels. Facing the inspection task of these 20,000 cameras, the required net processing time is approximately 208 minutes (about 3.5 hours).

[0080] This data fully demonstrates that the system of this invention, requiring only a single medium-configuration server, can easily complete the daily inspection of a network of tens of thousands of cameras within a few hours. Compared to the pure CPU solution described in the background art, which can only process about 35 video streams per minute (requiring about 9.5 hours to complete the same task), this invention improves efficiency by approximately 174%, achieving significant technological progress.

[0081] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A large-scale intelligent camera inspection system, characterized in that, include: The heterogeneous hardware platform is equipped with a central processing unit and a video accelerator card. The video accelerator card integrates a video processing unit and a neural network processing unit. The dynamic resource scheduling architecture, built on the heterogeneous hardware platform, consists of an application layer and an infrastructure layer. The application layer includes an inspection service module, a decoding module, and an AI analysis module. The inspection service is used to create tasks, synchronize status, and execute scheduling. The decoding module is used to call the video processing unit to perform hardware-level preprocessing on the video stream, converting the video stream into image data and writing the image data into shared memory. At the same time, it notifies the AI ​​analysis module through a lightweight RPC. The AI ​​analysis module is used to call the neural network processing unit to perform quality diagnosis on the preprocessed image data. The infrastructure layer uses Redis as a central event hub to receive task status feedback from the decoding module and the AI ​​analysis module.

2. The large-scale camera intelligent inspection system according to claim 1, characterized in that, The inspection service module includes a status manager, a backpressure controller, and a scheduler. The status manager is used to continuously synchronize the global status from the infrastructure layer and maintain the real-time status map of the system. The backpressure controller is used to execute the backpressure algorithm based on the global status data and decide whether to allow the allocation of new tasks. The scheduler is used to receive instructions from the backpressure controller and execute specific task allocation actions.

3. The large-scale intelligent camera inspection system according to claim 1, characterized in that, The AI ​​analysis module includes an RPC receiving thread pool and an AI analysis thread pool. The RPC receiving thread pool is used to receive RPC notifications from the decoding module and generate tasks to be analyzed into an internal central task queue. The AI ​​analysis thread pool is used to retrieve tasks from the central task queue and retrieve preprocessed image data from shared memory for quality diagnosis.

4. A large-scale intelligent inspection method using cameras, characterized in that, Includes the following steps: The inspection service module creates inspection tasks and assigns them to the decoding module. The decoding module retrieves video stream data from the inspection task and decapsulates it. It then calls the video processing unit to preprocess the video stream data and convert it into image data. The AI ​​analysis module receives image data, analyzes it to obtain diagnostic results, and publishes the diagnostic results to the central event hub. The inspection service module retrieves the diagnostic results from the central event hub and saves the diagnostic results to the business database.

5. The large-scale intelligent camera inspection method according to claim 4, characterized in that, The inspection service module continuously monitors the current effective load in the dynamic resource scheduling architecture through the back pressure controller. If the current effective load does not exceed the maximum concurrency of AI analysis, the task is assigned to the decoding module.

6. The large-scale intelligent inspection method for cameras according to claim 5, characterized in that, The current effective load is determined based on the number of tasks in decoding and the number of tasks waiting to be analyzed, while the maximum concurrency of AI analysis is determined based on the number of video acceleration cards.

7. The large-scale intelligent camera inspection method according to claim 4, characterized in that, The decoding module and AI analysis module publish the task status to the central event hub, and the inspection service module obtains the task status through the central event hub and updates the task status synchronously.

8. The large-scale intelligent inspection method for cameras according to claim 4, characterized in that, The decoding module writes the image data into shared memory and notifies the AI ​​analysis module through a lightweight RPC framework, while also passing the task ID and shared memory identifier.

9. The large-scale intelligent camera inspection method according to claim 4, characterized in that, Set a timeout period for the task during the decoding or analysis process. If the timeout period is exceeded during the decoding or analysis process, the process is considered to have failed, and the corresponding failed process is restarted.

10. The large-scale intelligent camera inspection method according to claim 4, characterized in that, The diagnostic results include device ID, fault type, timestamp, and confidence level.

Images