A low-cost anomaly detection method and system based on multi-channel cameras

Abstract

The present invention discloses a low-cost anomaly detection method and system based on multiple cameras, relating to the field of intelligent robotics. The method comprises: using a multi-camera acquisition module to collect images of the field of view in front of the multiple cameras in real time; using a lightweight abnormal target detection model to perform real-time anomaly detection on the collected images and output the detection results; based on the detection results, combining multi-camera hierarchical detection with a multi-field dual-frame detection strategy, dynamically switching detection modes, and calculating historical anomaly scores to achieve graded responses in low-risk mode, potential anomaly mode, and high-anomaly warning mode. The present invention achieves reasonable resource allocation and efficient and accurate anomaly processing, enabling rapid response and decision-making to abnormal events, reducing the losses and impacts caused by anomalies.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robot technology, and in particular to a low-cost anomaly detection method and system based on multiple cameras. Background Art

[0002] With the rapid development of artificial intelligence and robotics, robots are increasingly being used across various industries. In particular, in the security sector, intelligent patrol robots, with their autonomous perception and decision-making capabilities, have become a crucial tool for enhancing safety and security. In security applications, intelligent patrol robots, equipped with cameras, perform real-time anomaly detection in the external environment, including unusual pedestrian behavior and environmental anomalies. Anomaly detection often focuses on precise coverage of the forward field of view. However, while traditional single-camera systems can provide certain monitoring capabilities, they still suffer from significant field of view limitations. While narrow-field cameras can clearly capture distant targets, they struggle to maintain wide-area perception. While wide-angle cameras can cover the surrounding area, they significantly lose detail in distant objects. To compensate for these limited fields of view, an increasing number of robots are adopting multi-camera systems. These systems combine multiple narrow-field and wide-angle cameras to provide comprehensive monitoring at varying distances. This allows robots to clearly detect anomalies close up while also capturing unusual objects at a distance, providing more efficient support for security operations.

[0003] However, while multi-camera-based anomaly detection systems provide a wider field of view, they also bring many challenges:

[0004] First, multi-channel camera systems face challenges in cost and performance optimization in low-speed scenarios:

[0005] In low-speed scenarios, the high-cost FPD-Link / GMSL camera solutions used by some existing systems have a significant price-performance imbalance. For example, the robot disclosed in the prior art uses a SerDes interface to obtain 1280×720@30fps image data output by the GMSL camera for target detection. However, in low-speed scenarios, the degree of motion blur is greatly reduced, and mainstream deep learning models often use lower resolutions to maintain detection accuracy. The video bandwidth provided by the GMSL / FPD-Link solution exceeds actual demand, resulting in serious hardware resource idleness in low-speed scenarios. The high procurement cost of the camera directly increases the overall cost of the system, which is in conflict with the cost sensitivity generally required in low-speed scenarios. Therefore, how to build a "resolution-bandwidth-cost" balanced solution suitable for low-speed scenarios through sensor selection and hardware architecture reconstruction has become a key issue that needs to be addressed urgently.

[0006] Second, multi-channel camera systems face challenges in optimizing computing resources in anomaly detection scenarios:

[0007] In anomaly detection scenarios, the resource overload caused by parallel processing of multiple cameras is particularly prominent. Especially for devices with limited computing resources, real-time detection of multiple cameras consumes a significant amount of GPU resources. Even devices with abundant computing resources, such as the Jetson AGX Orin, still need sufficient computing resources to ensure the proper operation of other core algorithms, such as navigation and obstacle avoidance. Therefore, how to ensure the accuracy of anomaly detection algorithms while designing a lightweight model to support parallel detection of multiple cameras with minimal resource consumption has become a critical issue that needs to be addressed.

[0008] Third, multi-channel camera systems face the problem of invalid detection redundancy in anomaly detection scenarios:

[0009] In a multi-camera system, the probability of abnormal events occurring is extremely low. When using the traditional fixed-cycle multi-camera polling detection mechanism, the system needs to continuously perform high-frequency invalid calculations on scenes without abnormalities, which will lead to idle waste of computing resources and reduced computing efficiency. This redundant detection mechanism not only increases the resource burden of the system, but also limits the coordinated analysis capabilities of multiple cameras. Therefore, how to reduce the detection of scenes without abnormalities by each camera has become a problem that needs to be solved urgently.

[0010] Therefore, there is an urgent need to develop a low-cost anomaly detection method and system based on multi-channel cameras to meet the needs of low-speed anomaly detection scenarios and improve the overall performance of the system. Summary of the Invention

[0011] In order to solve the technical problems existing in the above-mentioned prior art, the present invention proposes a low-cost anomaly detection method and system based on multiple cameras to perform efficient and accurate inspection of abnormal situations in real time.

[0012] On the one hand, to achieve the above-mentioned purpose, the present invention provides a low-cost anomaly detection method based on multiple cameras, comprising:

[0013] The multi-channel camera acquisition module collects the field of view images in front of the locations of the multi-channel cameras in real time;

[0014] Perform real-time anomaly detection on the collected images through a lightweight anomaly target detection model and output the detection results;

[0015] Based on the detection results, combined with multi-camera hierarchical detection and multi-field dual-frame detection strategies, the detection mode is dynamically switched, and a hierarchical response of low-risk mode, potential abnormality mode and high abnormality warning mode is achieved by calculating historical anomaly scores.

[0016] Preferably, the layout of the multiple cameras includes a composite layout of at least one wide-angle camera and multiple narrow-distance cameras, and full coverage of the field of view in front of the camera position is achieved through AHD technology. A multi-channel AHD signal analysis unit is constructed through the XS9922B chip, and the virtual channel technology is used to output the multiple video streams to the main control platform.

[0017] Preferably, the multi-channel cameras are connected to the CSI module of the main control platform through the XS9922B chip to achieve low-latency capture of the original image data, and finally realize remote real-time video transmission through RTP encapsulation and UDP transmission.

[0018] Preferably, the lightweight abnormal target detection model is a YOLOv8n-ADL model, which uses GhostConv to replace traditional convolution and generates feature maps through identity mapping and cheap operations; introduces the LSKA attention mechanism in the SPPF module, captures multi-scale features through separable one-dimensional convolution kernels; uses the WaveletUnPool module to replace traditional upsampling, and uses wavelet filters to restore image details; uses the LSDECD detection head to share convolution parameters, and uses the MPDIoU loss function to optimize the target box overlap problem.

[0019] Preferably, the multi-camera hierarchical detection includes:

[0020] Based on the detection results of the wide-angle camera, determine whether to enable the narrow-angle camera;

[0021] Abnormal targets whose confidence exceeds a preset threshold are detected in a graded manner, and the confidence scores of the wide-angle camera and the narrow-distance camera are integrated to calculate the abnormal situation score, specifically:

[0022]

[0023] In the formula, score is the abnormality score, N c is the number of narrow-distance cameras, C n,i is the confidence set of high-confidence abnormal targets detected by the i-th narrow-range camera, c i C n,i The target confidence in C w is the confidence set of the wide-angle camera, c is C w The target confidence in .

[0024] Preferably, the multi-field dual-frame detection strategy is:

[0025] The wide-angle camera image and the narrow-distance camera image selected by polling are stitched into a dual-frame input image. The detection object is dynamically selected based on the time difference between the last detection of the narrow-distance camera, and the detected abnormal targets are traversed and the anomaly score is calculated.

[0026] The dynamic selection detection objects are as follows:

[0027] i=argmax({T i |i∈{1,…,N c}});

[0028] Where i is the narrow-distance camera number, T i N is the time difference between the narrow-distance camera i and the last call, c The number of narrow-distance cameras.

[0029] Preferably, before performing the multi-camera hierarchical detection and multi-field of view dual-frame detection strategy, it also includes: obtaining the mapping relationship between the wide-angle camera and the narrow-distance camera through a calibration module, wherein the calibration module maps the narrow-distance camera field of view to the unified coordinate system of the wide-angle camera through checkerboard calibration and perspective transformation matrix calculation, and determines the projection area of ​​the narrow-distance camera through mask matching.

[0030] Preferably, the historical anomaly score is calculated as:

[0031] F(x)=(1-α)*F(x-1)+α*score;

[0032] Where α is the weighting coefficient, score is the anomaly score, F(x) is the historical anomaly score of x at the current moment, and F(x-1) is the historical score of x-1 at the previous moment.

[0033] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a low-cost anomaly detection system based on multiple cameras, comprising:

[0034] Multi-channel camera acquisition module: used to collect the field of view images in front of the location of multiple cameras in real time;

[0035] Lightweight anomaly detection module: used to perform real-time anomaly detection on collected images using the improved YOLOv8n model and output the detection results;

[0036] Mode switching and exception handling module: used to dynamically switch detection modes based on the detection results, combined with multi-camera hierarchical detection and multi-field dual-frame detection strategies, and achieve graded responses of low-risk mode, potential abnormality mode and high abnormality warning mode by calculating historical abnormality scores.

[0037] Preferably, the system supports access to a maximum of 16 AHD cameras, and achieves single-channel encoding delay ≤ 9.5ms and eight-channel concurrent delay ≤ 45.4ms through the nvv4l2h264enc hardware accelerator.

[0038] Compared with the prior art, the present invention has the following advantages and technical effects:

[0039] 1. Multi-channel camera cost optimization in low-speed scenarios: This invention adopts a multi-channel low-cost AHD camera access solution. By combining wide-angle cameras and narrow-distance cameras, it can achieve full-range vision at different distances. Without sacrificing system performance, it reduces hardware costs and achieves the effect of cost reduction and efficiency improvement.

[0040] 2. Improve and lighten YOLOv8n for common pedestrian and environmental anomaly scenarios, ensuring efficient and accurate detection of anomalies in real time without occupying excessive computing resources.

[0041] 3. The present invention proposes a mode switching and exception handling module based on historical scores. This mechanism can intelligently switch between multi-view dual-frame detection and multi-camera hierarchical detection according to historical scores, efficiently detect and save resources in long-term low-risk situations; comprehensively screen potential anomalies to avoid missed detections, and issue timely warnings in high-abnormality situations; it realizes reasonable resource allocation and efficient and accurate handling of anomalies, effectively utilizes computing power, reduces invalid detection redundancy, can quickly respond to abnormal events and make decisions, and reduce the losses and impacts caused by anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0043] Figure 1 This is a flow chart of a low-cost anomaly detection method based on multiple cameras according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a multi-channel camera layout according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of multi-channel camera parameters according to an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a lightweight abnormal target detection model construction process according to an embodiment of the present invention;

[0047] Figure 5 This is a flowchart of multi-camera hierarchical detection according to an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of a multi-field dual-frame detection strategy according to an embodiment of the present invention;

[0049] Figure 7 This is a workflow diagram of a calibration module according to an embodiment of the present invention;

[0050] Figure 8 This is a flowchart of mode switching and exception handling based on historical scores according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] This paper proposes a low-cost anomaly detection method based on multiple cameras, such as Figure 1 ,include:

[0054] The multi-channel camera acquisition module collects the field of view images in front of the locations of the multi-channel cameras in real time;

[0055] Use the improved YOLOv8n model to perform real-time anomaly detection on the collected images and output the detection results;

[0056] Based on the detection results, combined with multi-camera hierarchical detection and multi-field dual-frame detection strategies, the detection mode is dynamically switched, and a hierarchical response of low-risk mode, potential abnormality mode and high abnormality warning mode is achieved by calculating historical anomaly scores.

[0057] For low-speed scenarios, this embodiment combines edge computing nodes such as Jetson AGX Orin to design a low-cost multi-channel camera device. This device obtains a full range of views at different distances, achieves full coverage of the 180° field of view in front, and reduces the hardware cost of the multi-channel camera system. For the processing of abnormal scenarios, a lightweight model is combined to quickly and accurately detect the inputs of multiple cameras. At the same time, a mode switching and exception handling method based on historical scores is adopted, and an efficient detection strategy suitable for operation in abnormal and non-abnormal modes is proposed. Mode switching and exception handling are performed based on the historical anomaly scores of the current system. While performing comprehensive detection on each camera, the number of invalid detections is significantly reduced.

[0058] Furthermore, the multi-channel camera layout includes a composite layout of at least one wide-angle camera and multiple narrow-distance cameras, which uses AHD technology to achieve full coverage of the field of view in front of the camera location, and uses the XS9922B chip to build a multi-channel AHD signal analysis unit, and uses virtual channel technology to multiplex multiple video streams into a single MIPI-CSI2 interface to output to the Jetson AGX Orin main control platform.

[0059] Multiple cameras are connected to the CSI module of the main control platform through the XS9922B chip to achieve low-latency capture of original image data. The nvv4l2h264enc hardware accelerator is used to achieve encoding of single-channel 1080P@25fps video in 9.5ms and 45.4ms for eight concurrent channels. Finally, remote real-time video transmission is achieved through RTP encapsulation and UDP transmission.

[0060] Specifically, if Figure 2-Figure 3 In this embodiment, the multi-channel camera uses IMX290 as the image sensor for the multi-channel AHD camera. Its 1080P@25fps AHD signal output capability, combined with its wide dynamic range and high sensitivity, can adapt to complex lighting environments. The system achieves full coverage of the 180° front field of view and long and short detection distances through a composite layout of a 180° wide-angle lens and multiple narrow-distance cameras.

[0061] The XS9922B chip is used to build a four-channel AHD signal analysis unit. A single XS9922B chip supports four channels of 1080P@30fps AHD video signal analysis (also supporting HDCCTV and CVBS protocols). Virtual Channel (VC) technology enables signal-level multiplexing of four independent video streams, which are aggregated and output via a four-lane MIPI TX interface (each supporting a maximum D-PHY transmission rate of 1.5Gbps) onto a single MIPI-CSI2 interface. The Jetson AGX Orin control platform supports up to 16 channels of video input via a 4x4-lane MIPI-CSI2 interface.

[0062] A high-performance video processing pipeline is built on the Jetson AGX Orin platform, connecting to the XS9922B's MIPI-CSI2 channel via the CSI module to achieve low-latency capture of raw image data. The VI module further processes the raw data and supports up to 16 channels of video parsing and transmission through a virtual channel management mechanism. The encoding process uses the nvv4l2h264enc hardware accelerator to achieve fast encoding of single-channel 1080P@25fps video in 9.5ms and eight-channel concurrent encoding in 45.4ms. Finally, through RTP encapsulation and UDP transmission, remote real-time video transmission is achieved, supporting remote real-time video acquisition and analysis.

[0063] The AHD solution enables efficient integration of up to 16 cameras in low-speed scenarios, significantly reducing the cost of multi-camera systems while ensuring scenario requirements. By combining at least one 180° wide-angle lens with multiple narrow-pitch cameras, the narrow-pitch cameras can collectively cover a 180° field of view. Fixed physical positions ensure that the narrow-pitch cameras maintain their relative position to the wide-angle cameras, achieving full coverage of the 180° field of view and near and far detection distances. In this embodiment, three narrow-pitch cameras are used.

[0064] Furthermore, a lightweight abnormal target detection model is the YOLOv8n-ADL model. The YOLOv8n-ADL model uses GhostConv to replace traditional convolution, generates feature maps through identity mapping and cheap operations, introduces the LSKA attention mechanism in the SPPF module, captures multi-scale features through separable one-dimensional convolution kernels, uses the WaveletUnPool module to replace traditional upsampling, and uses wavelet filters to restore image details, uses the LSDECD detection head to share convolution parameters, and adopts the MPDIoU loss function to optimize the target box overlap problem.

[0065] Specifically, if Figure 4 Based on the improvement of the YOLOv8n model, this embodiment proposes a lightweight anomaly target detection model, YOLOv8n-ADL (YOLOv8n-Anomaly Detect-Lightweight), which covers the steps of data collection and processing, model improvement and lightweighting, model training, and model performance evaluation.

[0066] Step 1: Data collection and processing is divided into data acquisition, data labeling, and data set division:

[0067] Data collection: Based on common public safety needs, this embodiment constructs an abnormal event dataset covering environmental anomalies and pedestrian behavior anomalies. The categories of environmental anomalies are: fire, smoke, and traffic accidents; pedestrian behavior anomalies mainly include falls, fights, and carrying weapons. Since there are relatively few abnormal situation datasets in reality, when making the dataset, the datasets on the Internet are crawled; especially for the pedestrian anomaly dataset, abnormal behavior videos collected on the Internet are used, and images are collected every 5 frames, which effectively solves the oversampling problem caused by redundant actions. Through a standardized sampling process, it is ensured that 2,500 high-quality samples are obtained for each type of abnormal scene. Finally, an abnormal event dataset of 15,000 RGB images is constructed.

[0068] Data annotation: Use LabelImg software to annotate 15,000 images in the dataset. The image annotations are saved in TXT file format. Six target categories are planned to be annotated: fire, smoke, traffic accident, fall, fight, and weapon-holding.

[0069] Dataset division: A stratified random sampling strategy was used to divide the dataset into a training set and a validation set in a 4:1 ratio. Finally, 12,000 training sets and 3,000 validation sets were obtained.

[0070] Step 2: Model improvement and lightweighting:

[0071] To address the problems of insufficient local feature capture and decreased multi-scale target sensitivity of the YOLOv8n baseline model in complex scenarios, and to further reduce the consumption of system resources, this embodiment improves and lightweights the YOLOv8n network.

[0072] First, GhostConv is used to replace the traditional convolution, and C3Ghost is used to replace the traditional C2f module, which reduces the number of model parameters without losing detection accuracy. Secondly, the LSKA attention mechanism is introduced in SPFF to efficiently acquire multi-scale features. Then, the WaveletUnPool module is used to replace the traditional upsampling module, which is beneficial for the model to restore image details in the upsampling link and improve the accuracy of small target recognition. Then, the LSDECD head is used to replace the detection head, which shares convolution parameters while reducing the number of model parameters. Finally, the MPDIoU loss function is used to improve the problem of pedestrian target frame overlap.

[0073] The specific modules are as follows:

[0074] Module 1, GhostConv module: GhostConv is used in Backbone layers 1, 3, 5, and 7. GhostConv is divided into two steps. First, standard convolution is used to generate the initial feature map:

[0075] Y = X × f + b;

[0076] Among them, X is the input feature, f is the convolution kernel, b is the bias constant, and Y is the output feature map.

[0077] Then, the feature map y′ of each channel of the output feature map Y i , using φ i,j To generate the Ghost feature map y′ i,j , and finally get the Ghost feature map:

[0078]

[0079] Among them, φ i,j It contains a unit / identity mapping, which is mapped s times, each s times contains an identity mapping, s-1 is a cheap operate operation, and finally the channel output result of m*s is obtained.

[0080] Module 2, C3Ghost module: The C3Ghost module is used in Backbone layers 2, 4, 6, and 8. This module contains the GhostBottleneck module. The GhostBottleneck module sequentially performs GhostConv, batch normalization (BN), SiLU activation function, and GhostConv, and adds the result to the original feature map, maintaining the representational expression ability while reducing the number of parameters.

[0081] Module 3, SPPF_LSKA Module: The SPFF module's ability to capture fine-grained information needs improvement, making it difficult to adapt to complex anomaly detection scenarios. Therefore, the LSKA attention module is introduced at Backbone's 9th layer. Combined with the original SPFF module, it effectively improves the backbone feature network's ability to extract features at multiple scales, thereby enhancing detection effectiveness. The SPPF_LSKA module decomposes the traditional two-dimensional weighted convolution kernel into two cascaded one-dimensional separable convolution kernels, significantly reducing computational complexity and memory requirements. It can capture a wider range of image features while maintaining efficient computation.

[0082] Module 4, the WaveletUnPool module, uses the WaveletUnPool upsampling module in layers 10 and 13 of Neck. It defines four 2×2 wavelet filters: low-frequency-low-frequency, low-frequency-high-frequency, high-frequency-low-frequency, and high-frequency-high-frequency. These filters are used to process smoothing, vertical and horizontal image details, and image detail. These wavelet filters are used to better restore image details through deconvolution (i.e., unpooling). This helps the model more accurately capture the details of small objects.

[0083] Module 5, LSDECD: In the Head, LSDECD is used to replace the native head of YOLO8n. LSDECD merges the original three feature extractions of the head into a shared convolution, and then uses the scale scaling module for scale scaling. All normalization layers in the convolution are replaced with Group Normalization (GN), thereby reducing the model parameters while ensuring that the model has sufficient positioning and classification performance.

[0084] Module 6, MPDIoU loss function, uses this loss function to reduce the problem of recognition performance degradation caused by pedestrian border overlap.

[0085] The MPDIoU calculation formula is as follows:

[0086]

[0087] Among them, d1 and d2 are the squares of the Euclidean distances from the upper left corner of the predicted bounding box to the upper left corner and lower right corner of the true bounding box, respectively; w and h represent the width and height of the true bounding box, respectively; IoU is the intersection over union (IoU) of the predicted box and the true box; MPDIoU can reduce the problem of recognition performance degradation caused by overlapping pedestrian borders.

[0088] Step 3: Model training:

[0089] This object recognition algorithm was implemented on an Ubuntu 20.04 LTS operating system using the PyTorch 2.0.0 and cu124 deep learning framework, Python 3.10.12, and an Intel(R) Core(TM) i5-13600KF @ 3.5GHz (16-core, 24-thread) processor and an NVIDIA GeForce RTX 4060Ti 8GB CDDR6 graphics card. The YOLOv8n-ADL model was trained for 300 epochs, using the SGD optimizer for gradient updates with a batch size of 32 and a learning rate of 0.001. The trained weights were saved as the file best.pt.

[0090] Model performance evaluation: To objectively evaluate the performance of the object detection algorithm, we use precision (P), recall (R), average precision (AP), parameter count (Param), floating-point operations per second (FLOPs), and model memory usage as evaluation metrics, as shown in Table 1.

[0091] Table 1

[0092]

[0093] Furthermore, multi-camera hierarchical detection includes:

[0094] Based on the detection results of the wide-angle camera, determine whether to enable the narrow-angle camera;

[0095] Abnormal targets whose confidence exceeds a preset threshold are detected in a graded manner, and the confidence scores of the wide-angle camera and the narrow-distance camera are integrated to calculate the abnormal situation score, specifically:

[0096]

[0097] In the formula, score is the abnormality score, N c is the number of narrow-distance cameras, C n,i is the confidence set of high-confidence abnormal targets detected by the i-th narrow-range camera, c i C n,i The target confidence in C w is the confidence set of the wide-angle camera, c is C w The target confidence in .

[0098] Specifically, multi-camera hierarchical detection involves the system synchronously capturing a frame of the wide-angle camera image and images from each narrow-pitch camera within the same acquisition cycle. Because the wide-angle camera maps to the corresponding narrow-pitch camera image, a hierarchical detection strategy can be used to mine the relevant information between the wide-angle and narrow-pitch camera images, accurately detecting anomalies.

[0099] like Figure 5 The specific steps of multi-camera hierarchical detection are as follows:

[0100] Step 1: Detect the wide-angle camera and determine whether to enable the narrow-angle camera for detection.

[0101] Using the YOLOv8n-ADL algorithm, the wide-angle camera is first used to detect abnormal targets. If a target is detected, the N detected targets are traversed to obtain the confidence score Conf of each target. j And the target box Box j (1≤j≤N).

[0102] For each target j, calculate whether it is within the narrow-range camera i (1≤j≤N c ) in the mapping field of view, where N c is the number of narrow-distance cameras, and the calculation formula is as follows:

[0103]

[0104] Among them, R i Indicates the mapping area of ​​the narrow-angle camera i in the wide-angle camera, InRegion j is the mapping area number where the jth target is located, N c is the number of narrow-distance cameras;

[0105] If InRegion j ≠0, it means that the jth target is located in the narrow-distance camera i, and the box area of ​​the jth target does not exceed the imaging field of view of the narrow-distance camera i. At this time, according to InRegion j The result and the confidence of the j-th target are used to determine whether to enable the narrow-range camera i for detection:

[0106]

[0107] Among them, Conf low is the low confidence threshold set, For narrow-distance camera i (i = InRegion j ) a flag indicating whether further testing is required;

[0108] When target j is in the imaging field of view of narrow-range camera i and the confidence level exceeds Conf low When If set to 1, further detection is required for the narrow-distance camera i to prevent missed detection of abnormal conditions.

[0109] Step 2: Save the abnormal target threshold of the wide-angle camera:

[0110] Traverse the N detected abnormal targets and classify them into two categories based on whether the target is in the imaging field of view of the narrow-range camera i and the confidence level:

[0111] Set C1 w : The target is not in the imaging field of view of the narrow-range camera i and the confidence level exceeds Conf low :

[0112] C1 w ={Conf j |j∈[1,N],InRegion j ≠0and Conf j >Conf low};

[0113] Collection C2 w : The target is in the imaging field of view of the narrow-range camera i and the confidence level exceeds Conf high :

[0114] C2 w ={Conf j |j∈[1,N],InRegion j ==0and Conf j >Conf high};

[0115] Merge set C1 w and C2 w , we get the confidence set C of all abnormal targets of the wide-angle camera w :

[0116] C w =C1 w ∪C2 w .

[0117] Step 3: Enable the narrow-range camera i for detection and save the results:

[0118] Traverse the abnormal flag Abnormal corresponding to each narrow-distance camera i i If Abnormal i ==1, then call YOLOv8n-ADL to detect the narrow-range camera i.

[0119] Traverse the detected N i Anomaly targets, record confidence And save the confidence level over Conf low Goal:

[0120]

[0121] Step 4: Calculate the anomaly score:

[0122] The anomaly score is obtained by combining the confidence set C of the wide-angle camera w The confidence values ​​of all elements in , the confidence set C of all narrow-distance cameras n,i The confidence values ​​of all elements in are accumulated and calculated as follows:

[0123]

[0124] In the formula, score is the abnormality score, N c is the number of narrow-distance cameras, C n,i is the confidence set of high-confidence abnormal targets detected by the i-th narrow-range camera, c i C n,i The target confidence in C w is the confidence set of the wide-angle camera, c is C w The target confidence in .

[0125] Furthermore, the multi-field dual-frame detection strategy is:

[0126] The wide-angle camera image and the narrow-distance camera image selected by polling are stitched into a dual-frame input image. The detection object is dynamically selected based on the time difference between the last detection of the narrow-distance camera, and the detected abnormal targets are traversed and the anomaly score is calculated.

[0127] The dynamic selection detection objects are as follows:

[0128] i=argmax({T i |i∈{1,…,N c}});

[0129] Where i is the narrow-distance camera number, Ti N is the time difference between the narrow-distance camera i and the last call, c is the number of narrow-pitch cameras.

[0130] Specifically, the multi-field-of-view dual-frame detection strategy was designed based on the input requirements of the YOLO model and the resolution characteristics of AHD images. It aims to optimize the utilization of image detection resources, ensure detection results, and achieve comprehensive and efficient detection of multi-camera data. Through its unique dual-frame detection strategy and multi-field-of-view fusion mechanism, it reduces image resource waste while simultaneously balancing wide-area fields of view and distant details for comprehensive detection.

[0131] The following is a further detailed explanation of its working principle:

[0132] Image resource optimization:

[0133] Since the YOLO model requires square images for detection, conventional AHD images (common resolutions are (1920, 1080) and (1280, 720)) need to be padded to adjust to a square image with an aspect ratio of 1:1 before being input into the model, which will cause a lot of image resource waste. It will be wasted after filling The double-frame detection strategy is to splice two frames of images up and down to form an image of size w×(2×h) and then feed it into the model, so that two frames of images can be detected at the same time in one detection, and only a small amount of time is wasted for each frame of image. This strategy can realize multi-camera detection with less system resource consumption.

[0134] Multi-field collaborative detection:

[0135] Wide-angle cameras capture a wide field of view, which is crucial for overall scene perception. The module continuously monitors the wide-angle camera, providing real-time insights into the entire monitored area and promptly detecting any anomalies. While ensuring continuous monitoring with the wide-angle camera, it also performs checks on the narrow-angle cameras in a strategically controlled manner. Narrow-angle cameras typically focus on specific areas or objects, providing more detailed information. This method of monitoring ensures that even distant anomalies are not missed.

[0136] like Figure 6 , the specific steps are as follows:

[0137] Step 1: During the same acquisition cycle, the system will synchronously capture one frame of image from the wide-angle camera and images from all narrow-angle cameras.

[0138] On the one hand, the dual-frame detection strategy will fixedly select the image of the wide-angle camera as the first frame, and on the other hand, it will be based on the time difference T between each narrow-angle camera and the last detection. i To determine, the selection formula is:

[0139] i=arg max({T i |i∈{1,…,N c}});

[0140] The above formula can be used to select narrow-distance cameras regularly and in a round-robin manner according to the time sequence of their last detection, thereby ensuring that each narrow-distance camera can be detected fairly.

[0141] Step 2: Call the YOLOv8n-ADL algorithm to detect the image after the double frame combination. Traverse the detected N d Anomaly target, record confidence as ConfDouble jd (1≤jd≤N d ) and save the target confidence level exceeding Conf low Goal:

[0142] C d ={ConfDouble jd |jd∈[1,N d ],ConfDouble jd >Conf low};

[0143] Step 3: Calculate the anomaly score:

[0144] The anomaly score is the confidence set C of the image after the two frames are combined d Elements are accumulated:

[0145]

[0146] In the formula, score is the abnormal score, cd is C d The target confidence in .

[0147] Furthermore, before performing multi-camera hierarchical detection and multi-field of view dual-frame detection strategies, it also includes: obtaining the mapping relationship between the wide-angle camera and the narrow-distance camera through a calibration module, wherein the calibration module maps the narrow-distance camera field of view to the unified coordinate system of the wide-angle camera through checkerboard calibration and perspective transformation matrix calculation, and determines the projection area of ​​the narrow-distance camera through mask matching.

[0148] Specifically, the calibration module acts during the system initialization phase. Since the relative physical positions of multiple narrow-distance cameras and wide-angle cameras remain unchanged, the mapping relationship between the wide-angle camera and each narrow-distance camera only needs to be obtained once during system initialization.

[0149] like Figure 7 , specifically including:

[0150] Step 1: Obtain the distortion coefficient of the wide-angle camera through checkerboard calibration, and perform dedistortion operation on the wide-angle camera. The field of view obtained after the dedistortion operation of the wide-angle camera is recorded as S0;

[0151] Step 2: Jointly calibrate each narrow-pitch camera and wide-angle camera. Use the calibration template to mark the field of view of narrow-pitch camera i as Si. Mark the four vertices of the narrow-pitch camera template and the corresponding four vertices of the wide-angle camera template. Use these corresponding vertices to obtain the perspective transformation matrix H from the narrow-pitch camera to the wide-angle camera:

[0152]

[0153] in, Represents linear transformations such as scaling, shearing, and rotation, [a 31 a 32 ] is used for translation, [a 13 a 23 ] T Used to generate perspective transformation, use this transformation matrix to map the image of the narrow-angle camera to the same viewing angle as the wide-angle camera:

[0154] [x ′ ,y ′ ,z′]=H[x,y,z];

[0155] Among them, x, y, z are the original coordinate values ​​of the narrow-distance camera, x ′ ,y ′ ,z′ is the coordinate value of the narrow-distance camera after perspective transformation.

[0156] Step 3: Calculate the projection area based on the template matching of the mask.

[0157] Since the area after perspective transformation is usually not rectangular, a mask is used to mark the background of the narrow-angle camera image after perspective transformation, and template matching with the mask is used to find the projection area of ​​the narrow-angle camera in the wide-angle camera and record it as Ri.

[0158] Furthermore, the historical anomaly score is calculated as:

[0159] F(x)=(1-α)*F(x-1)+α*score;

[0160] Where α is the weighting coefficient, score is the anomaly score, F(x) is the historical anomaly score of x at the current moment, and F(x-1) is the historical score of x-1 at the previous moment.

[0161] Specifically, if Figure 8 By defining the historical anomaly score formula, dynamic switching of detection modes and anomaly handling are achieved based on the anomaly historical data. The specific process is as follows:

[0162] Step 1: Use the calibration module to calibrate the wide-angle camera and each narrow-distance camera to obtain the projection area of ​​each narrow-distance camera on the wide-angle camera imaging image;

[0163] Step 2: To reasonably measure the historical abnormality of the system, define the historical abnormality score formula as follows:

[0164] F(x)=(1-α)*F(x-1)+α*score;

[0165] Among them, α is the weighting coefficient used to adjust the balance between historical scores and current scores. Under normal circumstances, to ensure that more emphasis is placed on the current detection situation, α is usually set to a value greater than 0.5. score is the anomaly score of this round output by both the multi-view dual-frame detection mode and the multi-camera hierarchical detection mode during operation. F(x) is the historical anomaly score of the current moment x, and F(x-1) is the historical score of the previous moment x-1.

[0166] Step 3: Acquire images from the wide-angle camera and each camera at the same time;

[0167] Step 4: Mode switching and handling strategies based on historical abnormal situations:

[0168] 1. Long-term low-risk mode switch: in F(x) <F low (F low ≈0), F low The low risk threshold for anomaly scores indicates that the system has been in a stable, non-abnormal state for a long time. In this case, even if occasional false detections occur, the system can quickly adjust based on its historical scores. In this case, the multi-view dual-frame detection mode is selected. This mode efficiently detects each camera and fully utilizes the efficiency and low resource consumption of the dual-frame detection strategy, ensuring detection while achieving efficient use of system resources.

[0169] 2. Switching to potential anomaly comprehensive detection mode: When F(x)>F low, it means that some abnormality may have occurred in the system. To comprehensively and accurately detect potential abnormalities and prevent missed detections, the module switches to multi-camera hierarchical detection mode. This mode uses a hierarchical detection method, starting with the wide-angle camera for preliminary screening, and then using the narrow-angle camera for detailed detection as needed, thus achieving comprehensive coverage and in-depth detection of the entire monitoring scene.

[0170] 3. High abnormality alarm and early warning mode processing: When F(x)>F high When F high If the anomaly score reaches a high risk threshold, the system is clearly experiencing a significant abnormality. In this case, not only will the local device trigger an abnormality alarm, but a warning message will also be sent to the remote server, along with a real-time video stream. This helps relevant personnel receive abnormality information immediately, enabling them to respond and make decisions quickly, minimizing the potential losses and impact of the abnormal event.

[0171] Step 5: Return to step 3.

[0172] This embodiment also provides a low-cost anomaly detection system based on multiple cameras, including:

[0173] Multi-channel camera acquisition module: used to collect the field of view images in front of the location of multiple cameras in real time;

[0174] Lightweight anomaly detection module: used to perform real-time anomaly detection on collected images using the improved YOLOv8n model and output the detection results;

[0175] Mode switching and exception handling module: used to dynamically switch detection modes based on the detection results, combined with multi-camera hierarchical detection and multi-field dual-frame detection strategies, and achieve graded responses of low-risk mode, potential abnormality mode and high abnormality warning mode by calculating historical abnormality scores.

[0176] The system supports up to 16 AHD cameras, and uses the nvv4l2h264enc hardware accelerator to achieve single-channel encoding delay ≤ 9.5ms and eight-channel concurrent delay ≤ 45.4ms.

[0177] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A low-cost anomaly detection method based on multiple cameras, characterized in that: include: The multi-channel camera acquisition module collects the field of view images in front of the locations of the multi-channel cameras in real time; Perform real-time anomaly detection on the collected images through a lightweight anomaly target detection model and output the detection results; Based on the detection results, combined with multi-camera hierarchical detection and multi-field dual-frame detection strategies, the detection mode is dynamically switched, and a hierarchical response of low-risk mode, potential abnormal mode and high abnormal warning mode is achieved by calculating historical anomaly scores; The multi-camera hierarchical detection includes: Based on the detection results of the wide-angle camera, determine whether to enable the narrow-angle camera; Abnormal targets whose confidence exceeds a preset threshold are detected in a graded manner, and the confidence scores of the wide-angle camera and the narrow-distance camera are integrated to calculate the abnormal situation score, specifically: ； Where, Score the anomaly. is the number of narrow-distance cameras, For the The confidence set of high-confidence abnormal targets detected by a narrow-range camera, for The target confidence in , is the confidence set of the wide-angle camera, for Target confidence in ; The multi-field dual-frame detection strategy is: The wide-angle camera image and the narrow-distance camera image selected by polling are stitched into a dual-frame input image. The detection object is dynamically selected based on the time difference between the last detection of the narrow-distance camera, and the detected abnormal targets are traversed and the anomaly score is calculated. The dynamic selection detection objects are as follows: ； Where, i Number the narrow-distance camera. For narrow-distance cameras i The time difference from the last call, is the number of narrow-distance cameras; The historical anomaly score is calculated as: ； Where, is the weighting coefficient, is the abnormality score, For the current moment Historical anomaly score, For the previous moment Historical scores.

2. The low-cost anomaly detection method based on multiple cameras according to claim 1, characterized in that: The multi-channel camera layout includes a composite layout of at least one wide-angle camera and multiple narrow-distance cameras. AHD technology is used to achieve full coverage of the field of view in front of the camera location, and a multi-channel AHD signal analysis unit is constructed through the XS9922B chip. Virtual channel technology is used to output multiple video streams to the main control platform.

3. The low-cost anomaly detection method based on multiple cameras according to claim 2, characterized in that: The multi-channel cameras are connected to the CSI module of the main control platform through the XS9922B chip to achieve low-latency capture of original image data, and finally realize remote real-time video transmission through RTP encapsulation and UDP transmission.

4. The low-cost anomaly detection method based on multiple cameras according to claim 1, characterized in that: The lightweight abnormal target detection model is the YOLOv8n-ADL model. The YOLOv8n-ADL model uses GhostConv to replace the traditional convolution and generates feature maps through identity mapping and cheap operations. The LSKA attention mechanism is introduced in the SPPF module to capture multi-scale features through separable one-dimensional convolution kernels. The WaveletUnPool module is used to replace the traditional upsampling and the wavelet filter is used to restore image details. The LSDECD detection head is used to share convolution parameters and adopt The loss function optimizes the target box overlap problem.

5. The low-cost anomaly detection method based on multiple cameras according to claim 1, characterized in that: Before performing the multi-camera hierarchical detection and multi-field of view dual-frame detection strategy, it also includes: obtaining the mapping relationship between the wide-angle camera and the narrow-distance camera through a calibration module, wherein the calibration module maps the narrow-distance camera field of view to the unified coordinate system of the wide-angle camera through checkerboard calibration and perspective transformation matrix calculation, and determines the projection area of ​​the narrow-distance camera through mask matching.

6. A low-cost anomaly detection system based on multiple cameras, characterized in that: include: Multi-channel camera acquisition module: used to collect the field of view images in front of the location of multiple cameras in real time; Lightweight anomaly detection module: used to perform real-time anomaly detection on collected images using a lightweight anomaly target detection model and output the detection results; Mode switching and exception handling module: used to dynamically switch detection modes based on the detection results, combined with multi-camera hierarchical detection and multi-field dual-frame detection strategies, and achieve graded responses of low-risk mode, potential abnormality mode and high abnormality warning mode by calculating historical abnormality scores; The multi-camera hierarchical detection includes: Based on the detection results of the wide-angle camera, determine whether to enable the narrow-angle camera; Abnormal targets whose confidence exceeds a preset threshold are detected in a graded manner, and the confidence scores of the wide-angle camera and the narrow-distance camera are integrated to calculate the abnormal situation score, specifically: ； Where, Score the anomaly. is the number of narrow-distance cameras, For the The confidence set of high-confidence abnormal targets detected by a narrow-range camera, for The target confidence in , is the confidence set of the wide-angle camera, for Target confidence in ; The multi-field dual-frame detection strategy is: The wide-angle camera image and the narrow-distance camera image selected by polling are stitched into a dual-frame input image. The detection object is dynamically selected based on the time difference between the last detection of the narrow-distance camera, and the detected abnormal targets are traversed and the anomaly score is calculated. The dynamic selection detection objects are as follows: ； Where, i Number the narrow-distance camera. For narrow-distance cameras i The time difference from the last call, is the number of narrow-distance cameras; The historical anomaly score is calculated as: ； Where, is the weighting coefficient, is the abnormality score, For the current moment Historical anomaly score, For the previous moment Historical scores.

7. The low-cost anomaly detection system based on multiple cameras according to claim 6, characterized in that: The system supports up to 16 AHD cameras, and uses the nvv4l2h264enc hardware accelerator to achieve single-channel encoding delay ≤ 9.5ms and eight-channel concurrent delay ≤ 45.4ms.

Images