A high-altitude projectile detection method based on active learning and neural network

Abstract

The application discloses a high-altitude projectile detection method based on active learning and a neural network, and relates to the technical field of target detection.The identification steps of the application are as follows: collecting high-altitude projectile falling object data, constructing an active target detection model, performing target tracking on a moving object, constructing an image classification neural network model, filtering interfering objects by using the image classification network, and excluding the motion trajectories of non-high-altitude projectiles.The application adopts an active learning method to select a small amount of samples for marking through a query strategy, so that the marked information is used to improve a prediction model; the classification network is used to filter interfering objects from the detection results through the training and classification results of the model, so that the problem of false detection of flying birds, flying insects and leaves in a high-altitude projectile scene can be solved, and the tracking cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method for detecting objects thrown from heights based on active learning and neural networks. Background Technology

[0002] With the further acceleration of urbanization in recent years, high-rise buildings have sprung up one after another, and the problem of objects being thrown from high-rise buildings has become increasingly serious.

[0003] The current common solution is to deploy security cameras with upward-facing lenses, and use dynamic target detection and tracking to filter out interfering objects thrown from heights.

[0004] Traditional methods for dynamic object detection in vision include frame differencing and optical flow detection, but both have many limitations. For example, frame differencing cannot avoid false detections of leaves, and many false detections occur even with slight camera shake. Optical flow detection suffers from similar problems, and its biggest drawback is that it is based on sparse feature point matching, thus it cannot effectively solve the problem of object segmentation. Although dense optical flow detection algorithms exist, they are time-consuming and have other issues, so they are not considered. Traditional tracking algorithms either rely on HOG features or feature points for tracking. However, because objects thrown from high altitudes are small and move very quickly, traditional feature-based methods cannot be used for tracking.

[0005] In addition, existing technologies also employ deep learning methods, deploying models on servers by collecting large amounts of training data. Supervised classification methods, represented by deep learning, do indeed outperform unsupervised methods such as background subtraction and optical flow. However, object detection based on YOLO and R-CNN series requires a large amount of prior data. Falling objects from high-rise buildings are numerous, and objects thrown from higher floors are often very small, making dataset collection difficult. These algorithms have poor generalization ability and high implementation costs. Furthermore, compared to unsupervised methods, they lack sufficient theoretical guarantees and have poor interpretability.

[0006] Existing patent 1, "A Method for Monitoring Objects Thrown from High Altitude", CN111539388A, compares the acquired images frame by frame, finds the image frame where the object first appears by timestamp, obtains the corresponding binarized image by binarization, and finally uses geometric methods, combined with the camera calibration position and the coordinates of the object, to calculate the final position of the object for tracking.

[0007] The existing patent has the following problems in target detection of moving objects: the design only proposes the comparison of consecutive frames and binarization image processing to detect objects thrown from high altitudes, without proposing a specific and effective projectile tracking model. From the technical perspective of target detection, we can consider that it can only perform preliminary moving target recognition and cannot effectively achieve subsequent acquisition of motion trajectories and dynamic target tracking of moving objects.

[0008] Existing patent 2, "A method, detection device and detection system for detecting moving targets thrown from high altitudes", CN202010598843.1, preprocesses the acquired image, uses inter-frame difference operation to obtain a difference image, then performs binarization processing and morphological image operations on the difference result, and finally performs connected component analysis on the morphological image to identify and track the target.

[0009] The existing patent 2 has the following problems in target tracking of moving objects: the technology uses an improved frame difference method and performs image analysis using morphological operations and connected components. Not only is the computational cost of calculating N difference images in a dataset of objects thrown from a height large, but analyzing the shape and size of moving objects solely based on connected regions still makes it difficult to achieve trajectory tracking of moving targets.

[0010] Existing patent three, "A Method, System, and Computer-Readable Medium for Identifying Objects Thrown or Falling from Heights," CN114639075A, first processes the monitoring image using an inter-frame difference method, then performs deduplication using an improved NMS algorithm to obtain the foreground detection object. Next, an improved SORT algorithm is used with Kalman filtering to predict the trajectory of the moving object, and finally, interfering objects are filtered out based on the laws of free fall.

[0011] The existing patent three has the following main problems in the process of filtering interfering objects. This patent proposes an improved NMS deduplication operation and an improved SORT algorithm, which can achieve dynamic target detection and target tracking to a certain extent. However, the limitation is that in the SORT algorithm, occlusion objects will have a high number of ID switches, which cannot effectively solve the occlusion tracking problem, and therefore cannot effectively filter interfering objects. Summary of the Invention

[0012] The purpose of this invention is to provide a method for detecting objects thrown from high altitudes based on active learning and neural networks. By selecting a small number of samples for labeling through a query strategy, the labeled information is used to improve the prediction model. This method is mainly designed for situations where there are few data labels for objects thrown from high altitudes. It can alleviate changes in detection information caused by partial occlusion and solve the problem of false detection of birds, insects, and leaves in high-altitude object throwing scenarios, thereby reducing tracking costs.

[0013] To achieve the above objectives, this invention provides a method for detecting objects thrown from heights based on active learning and neural networks, comprising the following steps:

[0014] S1. Collect data on falling objects from high altitudes: Collect and acquire video stream data between real buildings, extract frames from the video stream and decode it, and annotate the obtained moving object image frame data.

[0015] S2. Construct an active object detection model: Use a RetinaNet neural network with ResNet-50 as the base network as the object detector. Select samples for labeling through a query strategy based on active learning methods, and use the labeled information to improve the prediction model.

[0016] S3. Perform target tracking of moving objects: Use a dynamic target tracking algorithm for prediction, associate motion frame information through cascaded matching and IOU matching, use filters for prediction and updating, and obtain the tracking sequence number, motion trajectory and corresponding frame number information of the moving target.

[0017] S4. Construct an image classification neural network model: Deploy the image classification network model, obtain datasets from ImageNet and OpenImages, and classify interfering objects;

[0018] S5. Filtering interfering objects using an image classification network: Train and classify the neural network, apply the classification network model to the acquired moving target image, and filter out interfering objects from the output image based on the classification results of the CNN model.

[0019] S6. Eliminate non-high-altitude object throwing trajectories: Set filtering conditions for the trajectory characteristics of moving objects, and judge the trajectory of the object based on the target's moving distance and free fall law to eliminate abnormal motion trajectories in high-altitude object throwing.

[0020] Preferably, in step S1, collecting data on falling objects thrown from a height specifically includes the following steps:

[0021] S1-1. Obtain real video data through the camera to obtain an image containing RGB three-layer color channels;

[0022] S1-2, Perform video frame extraction and decoding to parse the video data into continuous image frame data;

[0023] S1-3. Label the category of moving targets in each frame of the image to form the category label {Yi, i = 1, ..., N} for that frame of the image.

[0024] Preferably, in step S2, constructing an active target detection model specifically includes the following steps:

[0025] S2-1, Annotate the image frame data of the moving object in step S1-2;

[0026] S2-2, Select a RetinaNet neural network with ResNet-50 as the base network as the basic detector;

[0027] S2-3. Input the labeled image frame data from step S2-1 into the active learning network framework;

[0028] S2-4. Define the Active Learner;

[0029] S2-5. Use the target detection model to obtain the bounding box of the moving target.

[0030] Preferably, defining the Active Learner in steps S2-4 specifically includes the following steps:

[0031] S2-4-1. Define model parameters and query strategies applied to request annotations;

[0032] S2-4-1-1, For the predicted bounding box B j The tightness is defined as follows;

[0033] T(B j ) = IOU(B j ,R j )

[0034] Among them, R j This refers to the candidate regions predicted by the network before adjustment.

[0035] S2-4-1-2, Based on the sample density T(B) j ) and confidence level P max Depending on the difference, the following selection strategy is used to update the bounding box;

[0036]

[0037]

[0038] S2-4-1-3. Calculate the stability S(B) of the current model to noise using the following formula. j );

[0039]

[0040] S2-4-1-4, If the stability S(B) j The value has reached a stable level, so there is no need to label the unlabeled images;

[0041] S2-4-1-5, For S(B) that did not reach a stable value in step S2-4-1-3 j The following formula is used to update the settings;

[0042]

[0043] S2-4-2. Using an uncertainty-based active learning method, the returned strategy selects the data that needs to be labeled.

[0044] S2-4-3. Determine whether the model accuracy meets the requirements, and use a query function to select the most useful data for manually labeling to improve the model accuracy;

[0045] S2-4-4: Add the data from step S2-4-3 to the training set samples according to the query criteria and query them in a loop.

[0046] Preferably, in step S3, target tracking of a moving object includes the following steps:

[0047] S3-1. Calculate the Intersection over Union (IOU) of the detection bounding box of the moving target at the current time t and the detection bounding box at the previous time.

[0048] S3-2. Determine whether the calculation result is less than the preset threshold. If so, consider the matching to have failed and perform Kalman filtering prediction.

[0049] S3-3. Initialize a new Kalman filter and record the tracking sequence number of the current moving target and the corresponding frame number information;

[0050] S3-4. Use a Kalman filter to predict the position of the moving target in the current frame, as well as the tracking box and detection box;

[0051] S3-5. Use a Kalman filter to predict the current state of the moving target based on the previous state, and calculate the information difference between the current frame and the next frame.

[0052] S3-6. Use a matching algorithm to perform cascade matching, and associate the tracking box in the current frame with the detection box in the next frame;

[0053] S3-7. Use Kalman filtering to update the predicted position based on the detection in step S3-2;

[0054] S3-8. Obtain the tracking trajectory of the moving object path-pred∈R tx4xN Simultaneously acquire motion image data of abnormally moving targets path-abpred∈R Tx4xN.

[0055] Preferably, step S4, which involves constructing an image classification network training model, specifically includes the following steps:

[0056] S4-1. Use ImageNet and Google OpenImages to obtain the dataset_one containing the interfering objects;

[0057] S4-2, Perform data preprocessing and data augmentation;

[0058] S4-3. Define a CNN model for image classification, consisting of 7 layers: layers 1, 3, and 5 are convolutional layers (Conv2D), layers 2 and 4 are pooling layers, and the last two layers are fully connected layers (Dense).

[0059] S4-3-1, Convolutional layer C1 uses six 5×5 convolutional kernels with a stride of 1 to perform convolution operations on the input layer, using the tanh activation function, and provides the input shape to the model, producing six 28×28 feature maps. one ;

[0060] S4-3-2, Pooling layer S2 uses max_pool for pooling with a stride of 2. This pooling reduces the data dimensionality, resulting in a pooling size of 2×2. After pooling, six 14×14 feature maps are obtained. two This serves as the input to the next layer of neurons;

[0061] S4-3-3, Convolutional layer C3 uses 16 convolutional kernels of size 5×5 with a stride of 1, and outputs 16 feature maps of size 10×10. three ;

[0062] S4-3-4, Pooling layer S4 uses max_pooling with a stride of 2 and a pooling size of 2×2, outputting 16 5×5 feature maps. four The number of neurons is reduced to 16×5×5=400;

[0063] In S4-3-5, convolutional layer C5 continues to use 5×5 convolutional kernels to convolve the output of layer S4. The number of convolutional kernels is increased to 120, the stride is 1, and the output is 120 1×1 feature maps. five ;

[0064] S4-3-6, fully connected layer F6 and C5 are fully connected, outputting 84 feature maps. six ;

[0065] S4-3-7, the fully connected layer F7 outputs a tensor of length 5, representing the interference object to which the extracted feature belongs. The interference objects are birds, insects, leaves, blankets, and clothes.

[0066] Preferably, in step S5, the process of sorting interfering objects using an image classification network specifically includes the following steps:

[0067] S5-1, Define training parameters and input;

[0068] S5-2, Train the neural network model;

[0069] S5-3, Prediction using neural network models;

[0070] S5-4. Apply an image classification network to classify the moving targets in step S3-9.

[0071] S5-5. If the output category is interfering objects such as birds, insects, or leaves, then filter and delete the image from the data image, retaining the moving targets in the high-altitude object throwing to avoid false detection and false alarm events.

[0072] Preferably, in step S6, excluding non-high-altitude projectile trajectories specifically includes the following steps:

[0073] S6-1. Select the trajectory based on the judgment conditions of objects thrown from a height, set the threshold of the vertical coordinate of the trajectory of the moving object to M, and the threshold of the number of tracking frames to T.

[0074] S6-2. Exclude motion trajectories that meet the judgment conditions of S6-1, and at the same time provide the position and trajectory information of the moving object in the high-altitude projectile.

[0075] Therefore, this invention employs the aforementioned high-altitude object throwing detection method based on active learning and neural networks. It selects a small number of samples for labeling through a query strategy, thereby utilizing the labeled information to improve the prediction model. This method is primarily designed to address the limited number of data labels for high-altitude object throwing. The dynamic target tracking algorithm predicts the bounding box obtained by the detector, updates the tracking box and detection box of the moving target using filters, and employs cascaded matching and matching association of motion frame information to mitigate changes in detection information caused by partial occlusion. The classification network filters out interfering objects from the detection results through model training and classification results, solving the problem of false detection of birds, insects, and leaves in high-altitude object throwing scenarios, thereby reducing tracking costs.

[0076] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0077] Figure 1This is a flowchart of a high-altitude projectile detection method based on active learning and neural networks according to the present invention;

[0078] Figure 2 This is a structural diagram of a high-altitude object identification system based on an active learning and neural network-based high-altitude object detection method of the present invention.

[0079] Figure 3 This is a structural diagram of an active target detection model for a high-altitude projectile detection method based on active learning and neural networks according to the present invention.

[0080] Figure 4 This is a flowchart of a dynamic target tracking algorithm for a high-altitude projectile detection method based on active learning and neural networks, according to the present invention.

[0081] Figure 5 This invention provides a training process for an image classification network model of a high-altitude projectile detection method based on active learning and neural networks.

[0082] Figure 6 This is a diagram of the convolutional neural network model structure of a high-altitude projectile detection method based on active learning and neural networks according to the present invention.

[0083] Figure 7 The diagrams show the implementation of a high-altitude projectile detection method based on active learning and neural networks, including leaf occlusion and false detection. a is a parabolic diagram of leaf occlusion; b is an active target detection diagram; c is a diagram of improved tracking results; d is a diagram of false leaf detection; e is a diagram of the applied image classification network; and f is a diagram of excluding false detections. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0085] This invention provides a method for detecting objects thrown from high altitudes based on active learning and neural networks, comprising the following steps:

[0086] S1. Collect data on falling objects thrown from a height.

[0087] S1-1. Obtain real video data through the camera to obtain an image containing RGB three-layer color channels;

[0088] S1-2, Perform video frame extraction and decoding to parse the video data into continuous image frame data;

[0089] S1-3. Label the category of moving targets in each frame of the image to form the category label {Yi, i = 1, ..., N} for that frame of the image;

[0090] S2. Construct an active target detection model

[0091] S2-1, Annotate the image frame data of the moving object in step S1-2;

[0092] S2-2, Select a RetinaNet neural network with ResNet-50 as the base network as the basic detector;

[0093] S2-3. Input the labeled image frame data from step S2-1 into the active learning network framework;

[0094] S2-4. Define the Active Learner;

[0095] S2-4-1. Define model parameters and query strategies applied to request annotations;

[0096] S2-4-1-1, For the predicted bounding box B j The tightness is defined as follows;

[0097] T(B j ) = IOU(B j ,R j )

[0098] Among them, R j This refers to the candidate regions predicted by the network before adjustment.

[0099] S2-4-1-2, Based on the sample density T(B) j ) and confidence level P max Depending on the difference, the following selection strategy is used to update the bounding box;

[0100]

[0101]

[0102] S2-4-1-3. Calculate the stability S(B) of the current model to noise using the following formula. j );

[0103]

[0104] S2-4-1-4, If the stability S(B) j The value has reached a stable level, and there is no need to label the unlabeled image.

[0105] S2-4-1-5, For S(B) that did not reach a stable value in step S2-4-1-3 j The following formula is used to update the settings;

[0106]

[0107] S2-4-2. Using an uncertainty-based active learning method, the returned strategy selects the data that needs to be labeled.

[0108] S2-4-3. Determine whether the model accuracy meets the requirements, and use a query function to select the most useful data for manually labeling to improve the model accuracy;

[0109] S2-4-4: Add the data from step S2-4-3 to the training set samples according to the query criteria and query them in a loop;

[0110] S2-5. Use the target detection model to obtain the bounding box of the moving target.

[0111] S3. Perform target tracking of moving objects.

[0112] S3-1. Calculate the Intersection over Union (IOU) of the detection bounding box of the moving target at the current time t and the detection bounding box at the previous time.

[0113] S3-2. Determine whether the calculation result is less than the preset threshold. If so, consider the matching to have failed and perform Kalman filtering prediction.

[0114] S3-3. Initialize a new Kalman filter and record the tracking sequence number of the current moving target and the corresponding frame number information;

[0115] S3-4. Use a Kalman filter to predict the position of the moving target in the current frame, as well as the tracking box and detection box;

[0116] S3-5. Use a Kalman filter to predict the current state of the moving target based on the previous state, and calculate the information difference between the current frame and the next frame.

[0117] S3-6. Use a matching algorithm to perform cascade matching, and associate the tracking box in the current frame with the detection box in the next frame;

[0118] S3-7. Use Kalman filtering to update the predicted position based on the detection in step S3-2;

[0119] S3-8. Obtain the tracking trajectory of the moving object path-pred∈R tx4xNSimultaneously acquire motion image data of abnormally moving targets path-abpred∈R Tx4xN .

[0120] S4. Construct an image classification neural network model

[0121] S4-1. Use the two largest image data sources, ImageNet and Google OpenImages, to obtain a dataset_one containing distracting objects such as birds, insects, and leaves;

[0122] S4-2, Perform data preprocessing and data augmentation;

[0123] S4-3. Define a CNN model for image classification, consisting of 7 layers: layers 1, 3, and 5 are convolutional layers (Conv2D), layers 2 and 4 are pooling layers, and the last two layers are fully connected layers (Dense).

[0124] S4-3-1, Convolutional layer C1 uses six 5×5 convolutional kernels with a stride of 1 to perform convolution operations on the input layer, using the tanh activation function, and provides the input shape to the model, producing six 28×28 feature maps. one ;

[0125] S4-3-2, Pooling layer S2 uses max pooling with a stride of 2 to reduce the data dimensionality. The pooling size is 2×2, resulting in six 14×14 feature maps. two This serves as the input to the next layer of neurons;

[0126] S4-3-3, Convolutional layer C3 uses 16 convolutional kernels of size 5×5 with a stride of 1, and outputs 16 feature maps of size 10×10. three ;

[0127] S4-3-4, Pooling layer S4 still uses max pooling with stride = 2, pooling size is 2×2, and outputs 16 5×5 feature maps. four The number of neurons is reduced to 16×5×5=400;

[0128] In S4-3-5, convolutional layer C5 continues to use 5×5 convolutional kernels to convolve the output of layer S4. The number of convolutional kernels is increased to 120, the stride is 1, and the output is 120 1×1 feature maps. five ;

[0129] S4-3-6, fully connected layer F6 and C5 are fully connected, outputting 84 feature maps. six ;

[0130] S4-3-7, Fully connected layer F7 is the output layer fully connected to fully connected layer F6, outputting a tensor of length 5, representing the interference objects to which the extracted features belong (the 5 categories are birds, insects, leaves, blankets, and clothes).

[0131] S5. Filtering interfering objects using an image classification network.

[0132] S5-1, Define training parameters and input;

[0133] S5-2, Train the neural network model;

[0134] S5-3, Prediction using neural network models;

[0135] S5-4. Apply an image classification network to classify the moving targets in step S3-9;

[0136] S5-5. If the output category is interfering objects such as birds, insects, or leaves, then filter and delete the image from the data image, retaining the moving targets in the high-altitude object throwing to avoid false detection and false alarm events.

[0137] S6. Excluding motion trajectories not caused by objects thrown from heights.

[0138] S6-1. Select the trajectory based on the judgment criteria for objects thrown from a height;

[0139] S6-1-1. Set the threshold value of the vertical coordinate of the trajectory of the moving object to M, and the threshold value of the number of tracking frames to T;

[0140] S6-1-2. Exclude moving targets whose vertical descent distance is greater than the threshold M;

[0141] S6-1-3. Exclude motion targets whose trajectories do not conform to free fall or projectile motion;

[0142] S6-1-4. Exclude moving targets whose tracking frame count is less than the preset threshold T;

[0143] S6-1-5. Exclude moving targets whose trajectory starts outside the monitoring boundary;

[0144] S6-2. Exclude motion trajectories that do not meet the above conditions, and provide the position and trajectory information of the moving object in the high-altitude projectile.

[0145] Example

[0146] Figure 1 This is a flowchart of a high-altitude projectile detection method based on active learning and neural networks according to the present invention; Figure 2This is a structural diagram of a high-altitude object identification system based on an active learning and neural network-based high-altitude object detection method of the present invention.

[0147] S1. Collect data on objects falling from a height. Specific operation steps include:

[0148] S1-1. Acquire real video data through a binocular camera to obtain an image containing RGB three-layer color channels;

[0149] S1-2, Perform video frame extraction and decoding processing to parse the video data into continuous image frame data;

[0150] S1-3. Label the category of moving targets in each frame of the image to form the category label {Yi, i = 1, ..., N} for that frame of the image;

[0151] S2, such as Figure 3 As shown, the specific steps for constructing an active object detection model include:

[0152] S2-1. Label the image data of the moving objects in step S1-2;

[0153] S2-2, Select a RetinaNet neural network with ResNet-50 as the base network as the basic detector;

[0154] S2-3. Use the labeled data from step S2-1 as input into the active learning network framework;

[0155] S2-4. Define the Active Learner;

[0156] S2-4-1. Define model parameters and query strategies applied to request annotations;

[0157] S2-4-1-1, For the predicted bounding box B j The tightness is defined as follows;

[0158] T(B j ) = IOU(B j ,R j )

[0159] Among them, R j This refers to the candidate regions predicted by the network before adjustment.

[0160] S2-4-1-2, Based on the sample density T(B) j ) and confidence level P max Depending on the difference, the following selection strategy is used to update the bounding box;

[0161]

[0162]

[0163] S2-4-1-3. Calculate the stability S(B) of the current model to noise using the following formula. j );

[0164]

[0165] S2-4-1-4, If the stability S(B) j The value has reached a stable value, which means that even if the input unlabeled image is corrupted by noise, the detection result will not change significantly, so there is no need to label the unlabeled image.

[0166] S2-4-1-5, For S(B) that did not reach a stable value in step S2-4-1-3 j The following formula is used to update the settings;

[0167]

[0168] S2-4-2. Using an uncertainty-based active learning method, the returned strategy selects the data that needs to be labeled.

[0169] S2-4-3. Determine whether the model accuracy meets the requirements, and use a query function to select the most useful data for manually labeling to improve the model accuracy;

[0170] S2-4-4: Add the data from step S2-4-3 to the training set samples according to the query criteria and query them in a loop;

[0171] S2-5. Apply the target detection model to obtain the bounding box of the moving target;

[0172] S3, such as Figure 4 As shown, the specific steps for tracking a moving object include:

[0173] S3-1. Establish the following vector X for the bounding box of the moving target extracted in step S2-5 to represent the target's state information.

[0174] X = [x, y, w, h, dx, dy] T

[0175] Where x represents the x-coordinate of the center point of the box, y represents the y-coordinate of the center point of the box, w and h represent the width and height of the box respectively, and dx and dy are the velocity changes of x and y, all initialized to 0;

[0176] S3-2. Calculate the Intersection over Union (IOU) of the detection bounding box of the moving target at the current time t and the detection bounding box at the previous time.

[0177] S3-3. Determine whether the calculation result is less than the preset threshold. If so, consider the matching to have failed and perform Kalman filtering prediction.

[0178] S3-4. Initialize a Kalman filter and record the tracking sequence number of the current moving target and the corresponding frame number information;

[0179] S3-5. Use a Kalman filter to predict the position of the moving target in the current frame, as well as the tracking box and detection box;

[0180] S3-6, Using a Kalman filter based on the previous state The current state of the moving target Make predictions and calculate the information difference between the current frame and the next frame;

[0181] State prediction is performed using the following equation, where u t F represents the acceleration vector of an object. t ∈R 6x6 Let B be the state transition matrix. t ∈R 6x1 To control the input matrix;

[0182]

[0183] Calculate the information difference between the current frame and the next frame, and let the covariance matrix ∑ represent the uncertainty of the system at time t. Use the following equation to predict the covariance matrix at the current time.

[0184]

[0185] The matrix Q represents the uncertainty of the system noise, and the values ​​on the diagonal are the two coordinates of the center point and the noise of the width and height, respectively.

[0186] S3-7. Use a matching algorithm to perform cascade matching, and associate the tracking box in the current frame with the detection box in the next frame;

[0187] S3-7-1. Based on appearance information, calculate the cosine distance cost matrix between track and detection;

[0188] S3-7-2. Based on Mahalanobis distance, by setting a large value, some unsuitable terms in the cost matrix are filtered out, thereby obtaining the cost cosine matrix.

[0189] S3-7-3. Execute the Hungarian algorithm to obtain the successfully matched index pairs, with the row index being the index of track and the column index being the index of detection;

[0190] S3-8. Use Kalman filtering to update the predicted position based on the detection in step S2-2;

[0191] S3-8-1. Based on the detection at the current moment, correct the track state associated with it;

[0192] Correcting the associated track state involves constructing the state vector Z of the detection at time t. t The mean error between the detection and the track is calculated using the following formula, and the mean vector of the track is... Mapped to the detection space, where H is the measurement matrix;

[0193]

[0194] S3-8-2, Through the filter gain matrix K t To predict the covariance matrix ∑ t and the observation matrix R;

[0195] Trade-off prediction covariance matrix ∑ t The state estimate and the observation matrix R are corrected using the following formula.

[0196] K t =∑ t H T (H∑ t H T +R) -1

[0197]

[0198] S3-8-3. For each successfully matched track, update it with its corresponding detection and delete the unmatched tracks and detections.

[0199] Handling unmatched tracks and detections by obtaining the covariance matrix Mapping to the detection space, update the noise distribution of the optimal estimate using the following formula;

[0200]

[0201] S3-9. Obtain the tracking trajectory of the moving object path-pred∈R tx4xNSimultaneously acquire motion image data of abnormally moving targets path-abpred∈R tx4xN ;

[0202] S4, such as Figure 5 The specific steps for constructing the image classification network training model include:

[0203] S4-1. Use the two largest image data sources, ImageNet and Google OpenImages, to obtain the dataset_one containing the interfering objects;

[0204] S4-2, Perform data preprocessing and data augmentation;

[0205] S4-2-1. Convert all input images to 32×32 size through size normalization;

[0206] S4-2-2, Divide the dataset into X_train, X_test, and X_test sets according to a ratio of 6:2:2 for training, validation, and test sets. Cross-Validation ;

[0207] S4-2-3. Set the input mean and sample mean of the dataset_one obtained in step S4-1 to 0;

[0208] S4-2-4. Divide the input image Y_image by the standard deviation of the dataset;

[0209] S4-2-5. Perform random image flipping and random image scaling operations within the range of 0-180°;

[0210] S4-3, Define a CNN model for image classification, such as Figure 6 As shown, there are a total of 7 layers in the neural network, of which layers 1, 3 and 5 are convolutional layers (Conv2D), layers 2 and 4 are pooling layers, and the last two layers are fully connected layers (Dense).

[0211] S4-3-1, Convolutional layer C1 uses six 5×5 convolutional kernels with a stride of 1 to perform convolution operations on the input layer. It uses the activation function tanh as shown in the following formula and provides the input shape to the model, generating six 28×28 feature maps. one ;

[0212] tanh(z)=2σ(2z)-1

[0213]

[0214] S4-3-2, Pooling layer S2 uses max pooling with a stride of 2 to reduce the data dimensionality. The pooling size is 2×2, resulting in six 14×14 feature maps. two This serves as the input to the next layer of neurons;

[0215] S4-3-3, Convolutional layer C3 uses 16 convolutional kernels of size 5×5 with a stride of 1, and outputs 16 feature maps of size 10×10. three ;

[0216] S4-3-4, Pooling layer S4 still uses max pooling with stride = 2, pooling size is 2×2, and outputs 16 5×5 feature maps. four The number of neurons is reduced to 16×5×5=400;

[0217] In S4-3-5, convolutional layer C5 continues to use 5×5 convolutional kernels to convolve the output of layer S4, increasing the number of kernels to 120, with a stride of 1, outputting 120 1×1 feature maps. five ;

[0218] S4-3-6, fully connected layer F6 and C5 are fully connected, outputting 84 feature maps. six ;

[0219] S4-3-7. The fully connected layer F7, i.e., the output layer, is fully connected to the fully connected layer F6, using a radial basis function (RBF) network connection method as shown in the following formula, where X is... p The p-th input sample, C i This represents the i-th center point;

[0220]

[0221] S4-4-8. From the RBF structure in step S4-3-7, the output of the network can be obtained as follows, where h is the number of nodes in the hidden layer, and the final output is a tensor of length 5, representing the interference objects to which the extracted features belong (the 5 categories are birds, insects, leaves, blankets, and clothes).

[0222]

[0223] S5. Filtering interfering objects using an image classification network, the specific steps of which include:

[0224] S5-1, Define training parameters and input;

[0225] S5-1-1. Set algorithm parameters, data dimensions, number of categories, and set batchsize = 1024;

[0226] S5-1-2, Define the dataset_one containing the disturbance motion from step S4-1 as the image input layer X_image and its class label;

[0227] S5-1-3. Convert the image into a matrix format of 32×32 input layer size, and normalize the matrix;

[0228] S5-2, Train the neural network model;

[0229] S5-2-1. Obtain the predicted value Y_train through forward propagation calculation;

[0230] S5-2-2. Use the backpropagation algorithm to optimize the parameter values ​​of the neural network;

[0231] S5-2-2-1. Define the loss function Softmax using the following formula, transforming the forward propagation result Y_train from step S5-1-4 into a probability distribution, where y1, y2, ... y n This is the output of the original neural network;

[0232]

[0233] S5-2-2-2, Use cross-entropy to define the loss function, and use the following formula to calculate the difference between the predicted probability distribution y and the true probability distribution y';

[0234]

[0235] S5-3. Use neural network models for prediction;

[0236] S5-3-1. Initialize prediction parameters;

[0237] S5-3-2, Define an input Label to fill the container, and set the dropout during training to 0.25;

[0238] S5-3-3, Use the moving target from step S3-9 as the input Y_input of the constructed CNN model;

[0239] S5-3-4. Define a label variable Train. When it is True, perform the training operation and save the trained model. When it is False, perform the prediction. Use 20% of the dataset for image classification prediction experiments.

[0240] S5-4. Apply an image classification network to classify the image Y_input obtained in step S5-3-3.

[0241] S5-5. If the output category label is a bird, insect, leaf, or other interfering object, then the image will be filtered out from the data image, while retaining the moving target in the high-altitude object throwing to avoid false detection and false alarm events.

[0242] S6. Excluding non-high-altitude object throwing trajectories, the specific operational steps include:

[0243] S6-1. Select the trajectory based on the judgment criteria for objects thrown from a height;

[0244] S6-1-1. Set the threshold value of the vertical coordinate of the trajectory of the moving object to M;

[0245] S6-1-2. The vertical coordinate of the moving target should be greater than the initial threshold and always have a downward trend;

[0246] S6-1-3. The trajectory of a moving target should roughly conform to free fall or projectile motion.

[0247] S6-1-4. The trajectory of a moving target should end at the edge of the camera.

[0248] S6-2. Exclude motion trajectories that do not meet the above conditions, and provide the position and trajectory information of the moving object in the high-altitude projectile.

[0249] Figure 7 The diagrams illustrate the implementation of a high-altitude projectile detection method based on active learning and neural networks, focusing on leaf occlusion and false detection. Figure a shows the parabolic image with leaf occlusion; figure b shows the active target detection; figure c shows the improved tracking results; figure d shows the false leaf detection; figure e shows the applied image classification network; and figure f shows the elimination of false detections. Figure 7 It can be seen that the present invention solves the problem of false detection of interfering objects in the scenario of objects being thrown from high altitudes.

[0250] This invention employs a high-altitude object throwing detection method based on active learning and neural networks. It selects a small number of samples for labeling through a query strategy, thereby utilizing the labeled information to improve the prediction model. This method is primarily designed to address the limited number of data labels for high-altitude object throwing. The dynamic target tracking algorithm of this invention predicts the bounding box obtained by the detector, updates the tracking box and detection box of the moving target using filters, and employs cascaded matching and matching association of motion frame information to mitigate changes in detection information caused by partial occlusion. The classification network proposed in this invention filters out interfering objects from the detection results through model training and classification results, solving the problem of false detection of birds, insects, and leaves in high-altitude object throwing scenarios, thereby reducing tracking costs.

[0251] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that they can still modify or make equivalent substitutions to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting objects thrown from high altitudes based on active learning and neural networks, characterized in that, Includes the following steps: S1. Collect data on falling objects from high altitudes: Collect and acquire video stream data between real buildings, extract frames from the video stream and decode it, and annotate the obtained moving object image frame data. S2. Constructing an active object detection model: A RetinaNet neural network with ResNet-50 as the base network is used as the object detector. Samples are selected for labeling through a query strategy based on an active learning method, and the labeled information is used to improve the prediction model. The specific steps include: S2-1. Label the image frame data of moving objects; S2-2, Select a RetinaNet neural network with ResNet-50 as the base network as the basic detector; S2-3. Input the labeled image frame data from step S2-1 into the active learning network framework; S2-4. Define the Active Learner, which includes the following steps: S2-4-1. Define model parameters and query strategies applied to request annotations, including the following steps: S2-4-1-1, For the predicted bounding box B j The tightness is defined as follows; T(B j )=IOU(B j ,R j ); Among them, R j This refers to the candidate regions predicted by the network before adjustment. S2-4-1-2, Based on the sample density T(B) j ) and confidence level P max Depending on the difference, the following selection strategy is used to update the bounding box; S2-4-1-3. Calculate the stability S(B) of the current model to noise using the following formula. j ); S2-4-2. Using an uncertainty-based active learning method, the returned strategy selects the data that needs to be labeled. S2-4-3. Determine whether the model accuracy meets the requirements, and use a query function to select the most useful data for manually labeling to improve the model accuracy; S2-4-4: Add the data from step S2-4-3 to the training set samples according to the query criteria and query them in a loop; S2-5. Apply the target detection model to obtain the bounding box of the moving target; S3. Perform target tracking of moving objects: Use a dynamic target tracking algorithm for prediction, associate motion frame information through IOU matching and cascaded matching, use filters for prediction and updating, and obtain the tracking sequence number, motion trajectory and corresponding frame number information of the moving target. S4. Construct an image classification neural network model: Deploy the image classification network model, obtain datasets through ImageNet and OpenImages, and classify interfering objects; S5. Filtering interfering objects using an image classification network: Train and classify the neural network, apply the classification network model to the acquired moving target image, and filter out interfering objects from the output image based on the classification results of the CNN model. S6. Eliminate non-high-altitude object throwing trajectories: Set filtering conditions for the trajectory characteristics of moving objects, and judge the trajectory of the object based on the target's moving distance and free fall law to eliminate abnormal motion trajectories in high-altitude object throwing.

2. The method for detecting high-altitude projectiles based on active learning and neural networks according to claim 1, characterized in that, In step S1, collecting data on falling objects thrown from a height specifically includes the following steps: S1-1. Obtain real video data through the camera to obtain an image containing RGB three-layer color channels; S1-2, Perform video frame extraction and decoding to parse the video data into continuous image frame data; S1-3. Label the category of moving targets in each frame of the image to form the category label {Yi, i = 1, ..., N} for that frame of the image.

3. The method for detecting high-altitude projectiles based on active learning and neural networks according to claim 1, characterized in that, In step S3, target tracking of the moving object is performed, including the following steps: S3-1. Calculate the Intersection over Union (IOU) of the detection bounding box of the moving target at the current time t and the detection bounding box at the previous time. S3-2. Determine whether the calculation result is less than the preset threshold. If so, consider the matching to have failed and perform Kalman filtering prediction. S3-3. Initialize a new Kalman filter and record the tracking sequence number of the current moving target and the corresponding frame number information; S3-4. Use a Kalman filter to predict the position of the moving target in the current frame, as well as the tracking box and detection box; S3-5. Use a Kalman filter to predict the current state of the moving target based on the previous state, and calculate the information difference between the current frame and the next frame. S3-6. Use a matching algorithm to perform cascade matching, and associate the tracking box in the current frame with the detection box in the next frame; S3-7. Use Kalman filtering to update the predicted position based on the detection in step S3-2; S3-8. Obtain the tracking trajectory of the moving object path-pred∈R tx4xN Simultaneously acquire motion image data of abnormally moving targets path-abpred∈R tx4xN .

4. The method for detecting high-altitude projectiles based on active learning and neural networks according to claim 1, characterized in that, In step S4, constructing the image classification network training model specifically includes the following steps: S4-1. Use ImageNet and Google OpenImages to obtain the dataset_one containing the interfering objects; S4-2, Perform data preprocessing and data augmentation; S4-3. Define a CNN model for image classification, consisting of 7 layers: layers 1, 3, and 5 are convolutional layers (Conv2D), layers 2 and 4 are pooling layers, and the last two layers are fully connected layers (Dense).

5. The method for detecting high-altitude projectiles based on active learning and neural networks according to claim 4, characterized in that, The specific steps of step S4-3 are as follows: S4-3-1, Convolutional layer C1 uses six 5×5 convolutional kernels with a stride of 1 to perform convolution operations on the input layer, using the tanh activation function, and provides the input shape to the model, producing six 28×28 feature maps. one ; S4-3-2, Pooling layer S2 uses max_pool for pooling with a stride of 2. This pooling reduces the data dimensionality, resulting in a pooling size of 2×2. After pooling, six 14×14 feature maps are obtained. two This serves as the input to the next layer of neurons; S4-3-3, Convolutional layer C3 uses 16 convolutional kernels of size 5×5 with a stride of 1, and outputs 16 feature maps of size 10×10. three ; S4-3-4, Pooling layer S4 uses max_pooling with a stride of 2 and a pooling size of 2×2, outputting 16 5×5 feature maps. four The number of neurons is reduced to 16×5×5=400; In S4-3-5, convolutional layer C5 continues to use 5×5 convolutional kernels to convolve the output of layer S4. The number of convolutional kernels is increased to 120, the stride is 1, and the output is 120 1×1 feature maps. five ; S4-3-6, fully connected layer F6 and C5 are fully connected, outputting 84 feature maps. six ; S4-3-7, the fully connected layer F7 outputs a tensor of length 5, representing the interference object to which the extracted feature belongs.

6. The method for detecting high-altitude projectiles based on active learning and neural networks according to claim 5, characterized in that: The interfering objects in step S4-3-7 are birds, insects, leaves, blankets, and clothes.

7. A method for detecting high-altitude projectiles based on active learning and neural networks according to claim 1, characterized in that, In step S5, the process of sorting interfering objects using an image classification network specifically includes the following steps: S5-1, Define training parameters and input; S5-2, Train the neural network model; S5-3, Prediction using neural network models; S5-4. Apply an image classification network to classify the moving targets in step S3-9; S5-5. If the output category is interfering objects such as birds, insects, or leaves, then filter and delete the image from the data image, retaining the moving targets in the high-altitude object throwing to avoid false detection and false alarm events.

8. A method for detecting high-altitude projectiles based on active learning and neural networks according to claim 1, characterized in that, In step S6, excluding motion trajectories that are not from high altitudes, the specific steps include: S6-1. Select the trajectory based on the judgment conditions of objects thrown from a height, set the threshold of the vertical coordinate of the trajectory of the moving object to M, and the threshold of the number of tracking frames to T. S6-2. Exclude motion trajectories that meet the judgment conditions of S6-1, and at the same time provide the position and trajectory information of the moving object in the high-altitude projectile.

Images