A method and system for specific target detection based on the combination of three-level sensors and vision.

Abstract

This invention provides a specific target detection method based on a combination of three-level sensors and vision, comprising: a first-level detector using sensor information to detect the presence of a moving target; if a moving target is present, a second-level detector using visual information to locate the target's position; and a third-level detector performing behavior recognition based on the target's position to detect the target's behavior. This invention can both supplement insufficient visual information with sensor information, avoiding missed detections due to weather or occlusion, and effectively analyze target behavior to locate specific targets, preventing normal targets from being mistakenly identified as specific targets.

Description

Technical Field

[0001] This invention relates to the field of specific target detection technology in surveillance scenarios, specifically to a specific target detection method, system, device, and medium based on a combination of three-level sensors and vision. Background Technology

[0002] Security surveillance technology focuses on the effective deployment of surveillance equipment and the use of security technologies to ensure the safety of various facilities and equipment, preventing intrusion and damage by unauthorized personnel. This often requires deploying surveillance cameras in key areas to detect the presence of targets within those areas. Simultaneously, to filter out interference from normal targets and locate the true specific target, target detection can be aided by behavior recognition to identify the target's behavior and determine whether it matches the behavioral patterns of a specific target.

[0003] In the field of object detection, significant breakthroughs have been achieved in statistical learning-based object detection algorithms in recent years, driven by the development of deep learning. Generally, statistical learning-based object detection algorithms can be categorized into traditional manual feature-based object detection algorithms and deep feature-based machine learning object detection algorithms.

[0004] Traditional hand-designed feature-based object detection algorithms primarily utilize manually designed features to model object detection. Recent high-performing hand-designed feature algorithms include: the DPM (Deformable Part Model) algorithm (Object detection with discriminatively trained part-based models) proposed by Pedro F. Felzenszwalb et al. in 2010; ICF (Integral Channel Features) proposed by Piotr Dollár et al. in 2009; and the ACF (Fast Feature Pyramids for Object Detection) algorithm proposed in 2014. The Informed Harr method (Informed Haar-like Features Improve Pedestrian Detection) proposed by Shanshan Zhang et al. in 2014 aims to extract more representative Harr features for training. While these hand-designed features have achieved some success, their detection accuracy remains low due to the insufficient representational power of hand-designed features. Deep convolutional neural network models, with their more powerful feature learning and representation capabilities, are increasingly widely and successfully applied in image object classification and detection. The basic object detection operator is the R-CNN (Region-Convolutional Neural Network) model. In 2014, Girshick et al. proposed R-CNN for general object detection, followed by Fast R-CNN and Faster R-CNN, which improved the accuracy and speed of deep learning-based object detection algorithms. Algorithms such as YOLO and SSD, proposed in 2016, achieve fast object detection in a single stage through ideas such as Anchor.

[0005] Target detection and recognition in the field of surveillance has wide practical applications in areas such as security monitoring and is also a popular research direction in computer vision. However, existing methods still face the following difficulties and challenges, and their detection results need further improvement:

[0006] How to avoid missing target detection at night, in obstructed conditions, and at long distances? In practical security monitoring applications, at night, due to insufficient lighting, targets are often difficult to distinguish from the background in the monitoring image, making it difficult for traditional target detection methods to extract effective information. When targets are obstructed or at a distance, only a portion of the target's features can be effectively extracted. Therefore, relying solely on visual information is often insufficient to maintain 24 / 7 monitoring of the monitored area, while sensors can compensate for the shortcomings of visual information from another perspective. Utilizing the fusion of sensor and visual information for decision-making can effectively improve the 24 / 7 coverage capability of the monitoring deployment.

[0007] Traditional object detection methods rely on appearance features to detect and classify various targets from still images, lacking the ability to analyze and model target behavior to determine whether a target is a specific object. Analyzing still images in isolation is insufficient to determine target actions; multi-frame image information needs to be fused and analyzed to filter out interference from normal targets and identify the true specific target.

[0008] To effectively identify a specific target, behavior recognition methods can be used to identify and analyze the actions performed by the target. Behavior recognition research methods can be mainly divided into two categories: representation-based methods and deep network-based methods.

[0009] Representation-based methods for behavioral representation fall into two categories: global representation, which extracts an overall representation of human structure, shape, and movement; and local representation, which mines local information from videos. Global representation utilizes the video's global motion energy image (MEI) and motion history image (MHI). MEI uses a binary image to represent the location of motion in the video, while MHI uses a grayscale image to represent changes in the motion image. Local representation methods typically follow this framework: detection of space-time interest points (STIPs); extraction of local descriptors; and fusion of local descriptors. Key methods include extending two-dimensional Harris detection to three dimensions; and Gabor filtering in both time and space dimensions to address the problem of insufficient STIPs. Furthermore, feature extraction of motion trajectories to capture local motion information has also attracted widespread attention. The Dense Trajectory (DT) algorithm performs dense sampling of each frame of video and tracks the trajectory of each sampled point, extracting trajectory feature descriptors. It also extracts HOG, HOF, and MBH features from the region near the trajectory and finally encodes these features for behavior classification. The Improved Dense Trajectory (iDT) algorithm improves upon DT by using projection transformations between consecutive frames to estimate camera motion, thus removing the impact of these transformations on trajectory estimation.

[0010] Deep network-based methods can be broadly categorized into three types: multi-stream network models, spatiotemporal network models, and human body information-based network models. Multi-stream network models utilize video to extract motion information such as optical flow, specifically extracting motion features from moving regions within the video. A typical method is the Two-Stream ConvNet, proposed by Simonyan et al. in 2014. This network treats the video as an image sequence, calculating CNN features for image frames using spatial flow and optical flow CNN features between several adjacent frames using temporal flow, finally fusing the two. Furthermore, the Temporal Segment Network (TSN) was proposed to address the poor performance of two-stream convNets on long video sequences. TSN networks are based on long-range temporal structure modeling, combining sparse temporal sampling strategies and video-level supervised learning to ensure more effective training across the entire video. Spatiotemporal network models utilize 3D convolutional networks to extract spatiotemporal features from video image sequences through 3D convolution, or to perform 2D convolution operations on each frame and then fuse the features from each frame. Typical methods include Convolutional 3D (C3D), which borrows the concept of 2D convolutional networks from the image domain, adding a temporal dimension to the convolutional kernels to perform convolution operations on temporally adjacent frames and extract temporal information. Inflated 3D ConvNet (I3D), proposed in 2017, explores how to set the receptive field in the temporal dimension, introducing the Inflated Inception module and extending it to three dimensions. It also applies the idea of ​​two-stream convolutional networks to 3D convolutional networks, combining RGB information with optical flow information. Human body information-based network models extract the skeleton joint positions of pedestrians in videos and use the skeleton information as features to assist in behavior recognition and classification. Human posture changes often vary significantly when performing different behaviors. Human skeleton-based networks utilize extracted human posture or skeleton information to perform spatiotemporal modeling of key human information. Many works introduce graph models to extract features from skeleton changes. For example, AS-GCN treats the joints of the skeleton as nodes in a graph network and uses behavioral connections and structural connections to characterize the correlation between nodes. ST-GCN models the skeleton at different times as a spatiotemporal graph network and extracts its spatiotemporal features through graph convolution. Potion describes the temporal changes in pose by coloring poses at different times.

[0011] However, the above-mentioned behavior recognition methods all rely on video feature extraction in a relatively isolated manner, without fully integrating object detection modules to focus on key target activity areas. In fact, using object detection as a pre-detector, and then performing behavior recognition on cropped and scaled regions, can effectively help the model directly focus on key behavior occurrence areas. Furthermore, traditional methods such as TSN directly assess features frame-by-frame and then average the results for feature mining between adjacent frames, lacking the ability to mine and utilize the temporal relationship between adjacent frames. Summary of the Invention

[0012] To address the shortcomings of existing technologies, the purpose of this invention is to provide a specific target detection method and system based on a combination of three-level sensors and vision.

[0013] According to one aspect of the present invention, a method for detecting a specific target based on a combination of a three-level sensor and vision is provided, comprising:

[0014] The first-level detector uses sensor information to detect the presence of a moving target;

[0015] If a moving target is present, the second-level detector uses visual information to locate the target's position.

[0016] The third-level detector performs behavior recognition and detects target behavior based on the target location.

[0017] Preferably, the first-level detector uses sensor information to detect the presence of a moving target, including:

[0018] Install vibration sensors and microwave sensors within the controlled area;

[0019] Monitor vibration and microwave signals within the controlled area;

[0020] The vibration signal is subjected to Fourier transform;

[0021] The microwave signal is processed using the Doppler effect.

[0022] The microwave signal and vibration signal are combined and then subjected to median filtering to determine whether a moving target exists.

[0023] Preferably, if a moving target is detected, an image sequence with a fixed frame rate is pushed to the second-level detector.

[0024] Preferably, the second-level detector uses visual information to locate the target position, including:

[0025] The image sequence with a fixed frame rate pushed by the first-level detector is used as the input of the second-level detector;

[0026] Target detection is performed on the image sequence to obtain the target bounding box positions;

[0027] The target bounding box position is cropped;

[0028] Standardize the dimensions of the cropped areas.

[0029] Preferably, if the proportion of images in the image sequence that detect the target exceeds a set threshold, then cropping is performed; if it is below the set threshold, then the image sequence is considered to not contain the specific target.

[0030] Preferably, the process of cropping the target location and standardizing the dimensions includes:

[0031] The target bounding boxes obtained from each frame of the image are summarized to obtain a set;

[0032] Traverse the target boxes in the set to obtain their maximum x-coordinate Xmax and minimum xmin, and maximum y-coordinate Ymax and minimum ymin;

[0033] Construct a rectangle R using (Xmin, Ymin) and (Xmax, Ymax) as the diagonal points of the target box;

[0034] Each frame of the image is expanded by a fixed size in the horizontal and vertical directions of the image, with the center coordinate point P of the rectangle R as the center, and the expanded area is cropped out.

[0035] The cropped area is scaled to a fixed size using bilinear interpolation.

[0036] Preferably, the behavior recognition includes:

[0037] The sequence of images output by the second-level detector is used as the input to the third-level detector;

[0038] The image sequence is fed into a two-dimensional convolutional network to extract frame-by-frame features;

[0039] A fixed proportion of the features extracted from each frame is selected for temporal shifting to achieve temporal fusion.

[0040] The behavior classification of each frame feature is performed using a fully connected layer of a neural network, and then averaged to obtain the behavior recognition result.

[0041] Preferably, the time-domain shift includes two modes: an offline mode and an online mode.

[0042] The offline mode fuses features from past and future frames into the current frame;

[0043] The online mode merges past frames into the current frame.

[0044] Preferably, the time-domain shift shown includes:

[0045] One-quarter of the features are selected from each frame of the image, and one-eighth of the features are forward-shifted, that is, one-eighth of the features from the previous frame are shifted to the current frame, and one-eighth of the features from the current frame are shifted to the next frame.

[0046] The remaining 1 / 8 of the features are shifted backward, that is, 1 / 8 of the features from the next frame are shifted to the current frame, and 1 / 8 of the features from the current frame are shifted to the previous frame.

[0047] According to a second aspect of the present invention, a specific target detection system based on a combination of a three-level sensor and vision is provided, comprising:

[0048] The first-level detector module uses sensor information to detect moving targets;

[0049] The second-level detector module uses visual information to locate the position of the moving target.

[0050] The third-level detector module analyzes the behavior of the moving target to determine whether a specific target is present.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The specific target detection method based on the combination of three-level detection sensing and vision provided by this invention can not only use sensor information to supplement insufficient visual information and avoid missed detection due to weather or occlusion, but also effectively analyze target behavior to locate specific targets and avoid misdetecting normal targets as specific targets. Attached Figure Description

[0053] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0054] Figure 1 This is a flowchart of the first-level moving target detection process in one embodiment of the present invention;

[0055] Figure 2 This is a flowchart of the target scale transformation process in the second-level detector in one embodiment of the present invention;

[0056] Figure 3 This is a flowchart of the third-level behavior detection in one embodiment of the present invention. Detailed Implementation

[0057] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0058] Based on the shortcomings of existing technologies, this invention constructs a three-level detection model that utilizes both sensor and visual information. Sensor information can be used to assist in the detection of specific targets under complex weather or occlusion conditions, while visual information is used to detect targets within the area and determine whether a target is a specific target based on its behavior.

[0059] This invention provides an embodiment of a specific target detection method based on a combination of a three-level sensor and vision, comprising:

[0060] S100, the first-stage detector uses sensor information to detect the presence of a moving target;

[0061] S200, if a moving target exists, the second-level detector uses visual information to locate the target's position;

[0062] The S300, a third-level detector, performs behavior recognition and detects target behavior based on the target's location.

[0063] To more accurately determine whether a moving target exists, the present invention provides a preferred embodiment to perform S100.

[0064] Vibration and microwave sensors are installed in the controlled area to perform initial detection of the presence of moving targets. The sensors monitor vibration and microwave signals within the area. Vibration signals are used to detect moving targets or vibrations caused by impacts in key areas. The sensors receive the vibration and microwave signals, perform time-series filtering, and fuse the signals to filter out potential noise interference and reduce false alarms. Only after multiple anomaly detections following filtering and fusion decisions is a moving target considered present. If a moving target is found, an image sequence is pushed to the second-level detector. The push is at a certain frame rate, preferably 8 frames per 2 seconds. Multiple anomaly detections are performed to filter out occasional false alarms caused by environmental disturbances. Specifically, in practice, a moving target can be determined to exist after 3-5 consecutive anomaly detections.

[0065] Furthermore, such as Figure 1 The diagram shown is a schematic representation of the process for detecting a moving target using a first-level detector in this preferred embodiment; specifically, it includes:

[0066] S101, Vibration and microwave sensors are installed in the control area to receive vibration signals and microwave signals. The vibration signals are used to detect whether there is movement, and the microwave signals are used to detect the distance and direction of the target movement.

[0067] S102 performs a Fourier transform on the vibration signal to examine its spectral characteristics and detect whether there are abnormal fluctuations corresponding to the target movement. After detecting an anomaly, a time-series filtering operation is performed. Only when an anomaly is detected multiple times consecutively is it judged as an anomaly.

[0068] S103, the microwave signal analyzes the target's speed and direction of motion by sending microwaves and receiving reflected signals, combined with the Doppler effect;

[0069] S104, The vibration signal and microwave signal are combined for median filtering to determine whether there is a moving target;

[0070] S105, if it is determined that there is a moving target in the area, the image sequence is sampled at a fixed frame rate, one frame every 250 milliseconds, and every 8 frames of images form a sequence, which is then pushed to the second-level detector.

[0071] In order to accurately locate the target position, the present invention provides a preferred embodiment to perform S200.

[0072] The image sequence from the first-level detector is fed into the second-level target detection network, using the image sequence captured from surveillance video as input. After input, frame-by-frame feature extraction is performed, and pedestrians and other specific targets are detected frame-by-frame to obtain a sequence of target bounding boxes. For each bounding box, the maximum and minimum values ​​of its x and y coordinates are taken to form a bounding circle, cropping the region and using it as input to the behavior recognition network. However, due to interference from factors such as nighttime and occlusion, the target detection rate in the image may not be 100%. Therefore, a certain percentage threshold can be set according to actual needs, such as 50%. If more than 50% of the images in the sequence detect a target, the image sequence is then pushed to the behavior recognition module.

[0073] Furthermore, such as Figure 2 The diagram shown illustrates the process of locating the target position using the second-level detector in this preferred embodiment. The target detection module uses the YOLOv5 model as the target detector, taking the image with the target position marked as the training set input to learn target features. The model utilizes a two-dimensional convolutional network and pooling layers to extract features, and then regresses the target bounding box position through a fully connected layer to obtain the position and confidence score of each target in the image. Specifically, this includes:

[0074] S201, extract the image sequence {I1,I2,…} from the surveillance video at a fixed frame rate, and send each frame of the image sequentially to the target detector to detect the targets present in the image, thereby obtaining the target bounding box sequence for each frame of the image. I1 corresponds to B1: {B11,B12,B13…}, and I2 corresponds to B2: {B21,B22,B23…}.

[0075] S202, if the target detection module detects targets in more than half of the images in the image sequence {I1,I2,…}, perform bounding box extraction: put all target boxes into set S: {B11,B12,B13,…,B21,B22,B23,…}, traverse the target boxes in S1 to obtain their maximum x-coordinate Xmax, minimum xmin, maximum y-coordinate Ymax, and minimum ymin;

[0076] S203, Expand horizontally and vertically around the bounding box: Construct a rectangle R with (Xmin, Ymin) and (Xmax, Ymax) as the diagonal points of the target bounding box. Expand the rectangle R horizontally and vertically around its center point P. Then, crop this area from all images in the image sequence. Specifically, for an input image with width W and height H, the horizontal expansion pixel range can be set to [W / 3, 2W / 3] and the vertical expansion pixel range can be set to [H / 3, 2H / 3]. For example, for an input image with a size of 1920*1080, the horizontal expansion pixel range can be set to [640, 1280] and the vertical expansion pixel range to [360, 720].

[0077] S203 uses bilinear interpolation to scale all cropped areas from the image, transforming them into fixed-size (H, W) regions. When the target bounding box is located in a corner of the image, if the expanded area exceeds the image size during horizontal and vertical scene expansion, the actual image boundary must be used as the limit. In this case, the size of the cropped region may not be consistent with the size when the target bounding box is in the center of the image. Therefore, to ensure the consistency of the cropped region size, further scaling is required to guarantee consistent output dimensions.

[0078] S204, uniformly sized cut areas are sent to the behavior recognition module for recognition.

[0079] Based on the above embodiments, the present invention provides a specific application embodiment of target location localization using a second-level detector. First, the original input image sequence {I1,I2} with a size of 1280*720 is subjected to target detection to obtain the target bounding box sequence B1: {{(120,50), (260,80)}, {(420,450), (560,580)}}, B2: {{(330,40), (520,150)}, {(210,610), (430,860)}}, where each target bounding box includes information about the two diagonal points, the upper left corner and the lower right corner. Place all target bounding boxes into set S: {{(120,50), (260,80)}, {(420,450), (560,580)}, {(330,40), (520,150)}, {(210,610), (430,860)}}. Iterate through the target bounding boxes in S to obtain their maximum x-coordinate of 860 and minimum x-coordinate of 50, and maximum y-coordinate of 860 and minimum y-coordinate of 40. Using (120,40) and (560,860) as the diagonal points of the target bounding boxes, construct a rectangle R with the center point of R at (340, 450). Extend the rectangle 360 ​​pixels horizontally and vertically outward from the center point of R. If the extension exceeds the image boundary, extend only to the image boundary, resulting in a rectangular area with the top-left and bottom-right diagonal points of {(0,90), (700,810)}. The region is cropped from {I1,I2} and then scaled to a 720*720 resolution. Finally, the cropped image {I1',I2'} is sent to the behavior recognition module.

[0080] To accurately identify behaviors, the present invention provides a preferred embodiment for executing S300, which performs behavior recognition. Specifically:

[0081] S301, the behavior recognition network feeds the image sequence frame by frame into a two-dimensional convolutional network for feature extraction to obtain the features of a single frame image.

[0082] S302, and performs temporal domain shifting operation on frame-by-frame features to fuse the spatial features of adjacent frames in the temporal domain;

[0083] S303 feeds the fused features into a fully connected layer of a neural network for behavior recognition and classification. The behavior classification results of all frames are averaged and voted to obtain the behavior classification of the segment. The specific target is identified by judging whether the behavior is abnormal.

[0084] In this embodiment, the behavior recognition module uses a TSM network as its basic framework and incorporates a temporal multi-scale module. First, the received image sequence is fed frame by frame into a two-dimensional convolutional network (e.g., ResNet) to extract image features. Then, a portion of the image features from each frame is selected for temporal shifting, allowing information from different frames to be fused. The temporal shifting operation includes two modes: offline and online. Offline mode fuses features from past and future frames into the current frame; online mode fuses information from past frames into the current frame.

[0085] In other embodiments of the present invention, target detection in security monitoring has the following problems:

[0086] Weather Impact: Footage captured by surveillance cameras is inevitably affected by weather, resulting in variations in image quality. In rainy, foggy, or nighttime conditions, the distinction between the target and background decreases, making it difficult for surveillance cameras to accurately capture targets. Environmental Occlusion: When environmental occlusion exists in the image, only a portion of the target's features can often be detected and extracted, potentially leading to missed detections using traditional object detection methods. Lack of Behavioral Analysis: Object detection relies on the target's appearance features to determine its location and category. However, pedestrians of the same category may belong to different categories of targets depending on the actions they are performing, potentially falling under specific target categories or normal targets.

[0087] The specific target detection method combining three-level detection sensing and vision proposed in the above embodiments of the present invention addresses the aforementioned problems by using a design that cascades three detectors with uniform scale. This method can effectively solve problems such as target occlusion and weather effects, and can also effectively analyze target behavior, thereby further subdividing the target.

[0088] The method proposed in the above embodiments of the present invention includes the construction of a joint detection of a sensor detector, a target detector and a behavior detector; the entire detection process includes four stages, and the detection of a specific person is used as an example for application.

[0089] First, vibration and microwave sensors installed within the area are used to monitor vibration and microwave signals, serving as the first-level detector. Upon detecting multiple consecutive abnormal signals, a moving target is identified, and the monitoring camera is activated to send an image sequence of 8 frames per 2 seconds to the second-level detector. Specific operations are as follows... Figure 1 As shown.

[0090] 2. Target detection, including pedestrians and specific objects, is performed using a YOLO object detector. Each frame of image is fed into the YOLO model, where image features are extracted via a 2D convolutional network. Based on these features, the specific location of the bounding box and the target category are regressed.

[0091] 3. Cropping the target region and performing a scale-uniform transformation. Each frame of the image yields a target bounding box sequence I1 corresponding to B1: {B11, B12, B13…}, and I2 corresponding to B2: {B21, B22, B23…}. All target bounding boxes are placed into a set S: {B11, B12, B13,…, B21, B22, B23,…}. The target bounding boxes in S1 are traversed to obtain their maximum and minimum horizontal coordinates Xmax and Ymin, and maximum and minimum vertical coordinates Ymax and Ymin. Using (Xmin, Ymin) and (Xmax, Ymax) as the diagonal points of the target bounding box, a rectangular bounding box R is constructed. This region is cropped from all images in the image sequence. Using the rectangular bounding box R as the center, a 720-pixel region is extended horizontally and vertically outwards from the image to supplement scene information. A scale-uniform transformation is then performed to form a fixed-size (720, 720) region, which is then fed into the behavior recognition module as input. Specific operations are as follows: Figure 2 As shown.

[0092] IV. Behavior recognition in the target region is performed using a behavior recognition module. Taking the (720, 720) region output by the target detection as input, the image sequence is processed through a two-dimensional convolutional network and a fully connected layer to extract features. Each frame yields 256-dimensional features. Then, 1 / 4 of the features from each frame are selected for temporal shifting. One-eighth of the features are shifted forward (one-eighth of the features from the previous frame is shifted to the current frame, and one-eighth of the features from the current frame is shifted to the next frame); the other one-eighth of the features are shifted backward (one-eighth of the features from the next frame is shifted to the current frame, and one-eighth of the features from the current frame is shifted to the previous frame). Specifically, the missing parts of the feature vectors after shifting in the first and last frames are set to 0 (zero-padding). Then, behavior classification is performed using the 256-dimensional features of each frame. Each frame corresponds to a probability value for behavior classification. The probability values ​​of behavior classification across all frames are averaged, and the behavior with the highest probability is taken as the classification result. The classification results are compared with the labels, the loss value is calculated and backpropagated to train the behavior recognition model. Specific operations are as follows: Figure 3 As shown.

[0093] The loss calculation, as shown in the formula, is a loss function based on cross-entropy, used to determine the difference between the recognition result and the true result.

[0094]

[0095] Where M is the number of categories, y c p is a sign function that takes the value 1 when the true class of the sample equals c, and 0 otherwise. c This represents the probability that the model classifies the sample as class c.

[0096] In this embodiment of the invention, a specific target refers to a pedestrian performing abnormal behavior within the monitored area, as well as a specific abandoned object. The detection of specific abandoned objects can be performed directly using target detection methods, while a three-level detector combining sensing and vision can be used to detect pedestrians performing specific behaviors.

[0097] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A method for detecting specific targets based on a combination of three-level sensors and vision, characterized in that, include: The first-level detector uses sensor information to detect the presence of a moving target; If a moving target is present, the second-level detector uses visual information to locate the target's position. The third-level detector performs behavior recognition and detects target behavior based on the target location; The first-level detector uses sensor information to detect the presence of a moving target, including: Install vibration sensors and microwave sensors within the controlled area; Monitor vibration and microwave signals within the controlled area; The vibration signal is subjected to Fourier transform; The microwave signal is processed using the Doppler effect. The microwave signal and vibration signal are combined and then subjected to median filtering to determine whether a moving target exists. The second-level detector uses visual information to locate the target position, including: The image sequence with a fixed frame rate pushed by the first-level detector is used as the input of the second-level detector; Target detection is performed on the image sequence to obtain the target bounding box positions; The target bounding box position is cropped; Standardize the dimensions of the cropped areas; The behavior recognition includes: The sequence of images output by the second-level detector is used as the input to the third-level detector; The image sequence is fed into a two-dimensional convolutional network to extract frame-by-frame features; A fixed proportion of the features extracted from each frame is selected for temporal shifting to achieve temporal fusion. The behavior classification of each frame feature after fusion is performed by using a fully connected layer of a neural network, and then the average is calculated to obtain the behavior recognition result. The time-domain shift includes two modes: offline mode and online mode. The offline mode fuses features from past and future frames into the current frame; The online mode merges past frames into the current frame.

2. The specific target detection method based on the combination of three-level sensors and vision according to claim 1, characterized in that, If a moving target is detected, a fixed frame rate image sequence is pushed to the second-level detector.

3. The specific target detection method based on the combination of three-level sensors and vision according to claim 1, characterized in that, If the proportion of images in the image sequence that detect the target exceeds a set threshold, then cropping is performed; if the proportion is below the set threshold, then the image sequence is considered to not contain the specific target.

4. The specific target detection method based on the combination of three-level sensors and vision according to claim 1, characterized in that, The cropping target location, and the scaling, includes: Summarize the bounding boxes obtained from each frame of the image to obtain a set; Traverse the target boxes in the set to obtain their maximum x-coordinate Xmax, minimum xmin, maximum y-coordinate Ymax, and minimum ymin; Construct a rectangle R using (Xmin, Ymin) and (Xmax, Ymax) as the diagonal points of the target box; Each frame of the image is centered on the center coordinate point P of the rectangular frame R, and is expanded by a fixed size in the horizontal and vertical directions of the image to crop out the expanded area; The cropped area is scaled to a fixed size using bilinear interpolation.

5. The specific target detection method based on the combination of three-level sensors and vision as described in claim 1, characterized in that, The time-domain shift shown includes: One-quarter of the features are selected from each frame of the image, and one-eighth of the features are forward-shifted, that is, one-eighth of the features from the previous frame are shifted to the current frame, and one-eighth of the features from the current frame are shifted to the next frame. The remaining 1 / 8 of the features are shifted backward, that is, 1 / 8 of the features from the next frame are shifted to the current frame, and 1 / 8 of the features from the current frame are shifted to the previous frame.

6. A specific target detection system based on a combination of three-level sensors and vision, used to implement the specific target detection method according to any one of claims 1-5, characterized in that, include: The first-level detector module uses sensor information to detect moving targets; The second-level detector module uses visual information to locate the position of the moving target. The third-level detector module analyzes the behavior of the moving target to determine whether a specific target is present.

Images