A method and device for detecting abnormal targets based on images taken by unmanned aerial vehicles

Abstract

The present invention provides a method and device for detecting abnormal targets based on images taken by a drone, and relates to the field of artificial intelligence technology. The method comprises: obtaining an image taken by a drone containing a target to be detected, and calculating a first characteristic distance between two adjacent frames of images based on a first branch twin neural network; if it is determined that the first characteristic distance is greater than a first preset distance threshold, then calculating a second characteristic distance between two adjacent frames of images and a normal sample image respectively based on a second branch twin neural network; if it is determined that there is at least one target frame image whose second characteristic distance is greater than a second preset distance threshold, then determining that the target to be detected in the target frame image is an abnormal target. The device executes the above method. The method and device for detecting abnormal targets based on images taken by a drone provided in an embodiment of the present invention can realize abnormal target detection by a drone in real time, efficiently and accurately.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for detecting abnormal targets based on images taken by unmanned aerial vehicles. Background Art

[0002] With the development of drone technology, drones have been widely used in power systems, industry, agriculture, security and other fields. Especially in the field of inspection, the amount of video data collected by drone cameras in real time is huge, which contains a lot of useful information. How to quickly and accurately find abnormal targets in real-time video data, such as abnormal cable damage, has become a major problem in the field of drone inspection.

[0003] Most existing anomaly detection methods rely on manual observation or complex image anomaly detection algorithms. Manual observation is not only inefficient, but also prone to missed detection or false alarms. Anomaly detection technology based on video frame images is not only difficult to guarantee real-time requirements, but also requires a large amount of computing resource overhead. Summary of the invention

[0004] In view of the problems in the prior art, an embodiment of the present invention provides a method and device for detecting abnormal targets based on images taken by a drone, which can at least partially solve the problems in the prior art.

[0005] On the one hand, the present invention proposes a method for detecting abnormal targets based on images taken by a drone, comprising:

[0006] Obtain an image containing a target to be detected taken by a drone, and calculate a first feature distance between two adjacent frames of the image based on a first-branch twin neural network;

[0007] If it is determined that the first characteristic distance is greater than a first preset distance threshold, calculating the second characteristic distances between two adjacent frame images and the normal sample image respectively based on the second branch twin neural network;

[0008] The first branch twin neural network and the second branch twin neural network are respectively trained according to corresponding contrast loss functions and using corresponding training data sets; the contrast loss function uses the square term of the Euclidean distance between features of samples as a factor to enhance the distance difference and smooth the optimization process;

[0009] If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target.

[0010] The abnormal target detection method based on the image taken by the drone also includes:

[0011] If it is determined that the first feature distance is less than or equal to the first preset distance threshold, continue to calculate the first feature distance between the subsequent two adjacent frame images based on the first branch twin neural network traversal.

[0012] The abnormal target detection method based on the image taken by the drone also includes:

[0013] The first characteristic distance between two adjacent frames of images is calculated based on the first branch of the twin neural network, and the second characteristic distance between the two adjacent frames of images and the normal sample image is calculated based on the second branch of the twin neural network in parallel.

[0014] Wherein, pre-training the first branch twin neural network and the second branch twin neural network includes:

[0015] Acquire historical video data shot by a drone, wherein the historical video data contains the target to be detected;

[0016] Constructing a training data set according to the historical video data, wherein the training data set includes a first training data set consisting of continuous frame similar samples and continuous frame dissimilar samples, and a second training data set consisting of normal samples and abnormal samples;

[0017] Construct the first-branch twin neural network and the second-branch twin neural network, and use the first training data set to train the first-branch twin neural network until the first-branch twin neural network converges, and use the second training data set to train the second-branch twin neural network until the second-branch twin neural network converges.

[0018] Wherein, constructing the first branch twin neural network and the second branch twin neural network includes:

[0019] A detail feature extraction layer is constructed based on a deconvolutional neural network, and a global feature capture layer is constructed based on a TransformerEncoder with a single-head attention mechanism. The detail feature extraction layer and the global feature capture layer are connected in series to obtain the first-branch twin neural network and the second-branch twin neural network.

[0020] The step of training the first branch twin neural network using the first training data set includes:

[0021] The first branch Siamese neural network is trained using the first training data set according to the following first contrast loss function:

[0022]

[0023] Wherein, L1 is the first contrast loss function, Y1 is the label of the continuous frame sample pair, , is to balance the weights between similar samples and dissimilar samples, m1 is the first preset distance threshold, and D1 is the Euclidean distance of features between similar samples and dissimilar samples.

[0024] The step of training the second branch twin neural network using the second training data set includes:

[0025] The second branch Siamese neural network is trained using the second training data set according to the following second contrast loss function:

[0026]

[0027] Among them, L2 is the second contrast loss function, Y2 is the label of the normal sample and the abnormal sample pair, , is to balance the weight between normal samples and abnormal samples, m2 is the second preset distance threshold, and D2 is the Euclidean distance of the features between normal samples and abnormal samples.

[0028] On the one hand, the present invention proposes an abnormal target detection device based on images taken by a drone, comprising:

[0029] An acquisition unit, used for acquiring an image containing a target to be detected taken by a drone, and calculating a first characteristic distance between two adjacent frames of images based on a first branch twin neural network;

[0030] A calculation unit, configured to calculate, based on a second branch twin neural network, second characteristic distances between two adjacent frame images and a normal sample image if it is determined that the first characteristic distance is greater than a first preset distance threshold;

[0031] The first branch twin neural network and the second branch twin neural network are respectively trained according to corresponding contrast loss functions and using corresponding training data sets; the contrast loss function uses the square term of the Euclidean distance between features of samples as a factor to enhance the distance difference and smooth the optimization process;

[0032] The detection unit is used to determine that the target to be detected in the target frame image is an abnormal target if it is determined that there is at least one target frame image whose second characteristic distance is greater than a second preset distance threshold.

[0033] In another aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0034] The processor and the memory communicate with each other via the bus;

[0035] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the following method:

[0036] Obtain an image containing a target to be detected taken by a drone, and calculate a first feature distance between two adjacent frames of the image based on a first-branch twin neural network;

[0037] If it is determined that the first characteristic distance is greater than a first preset distance threshold, calculating the second characteristic distances between two adjacent frame images and the normal sample image respectively based on the second branch twin neural network;

[0038] The first branch twin neural network and the second branch twin neural network are respectively trained according to corresponding contrast loss functions and using corresponding training data sets; the contrast loss function uses the square term of the Euclidean distance between features of samples as a factor to enhance the distance difference and smooth the optimization process;

[0039] If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target.

[0040] An embodiment of the present invention provides a non-transitory computer-readable storage medium, including:

[0041] The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the following method:

[0042] Obtain an image containing a target to be detected taken by a drone, and calculate a first feature distance between two adjacent frames of the image based on a first-branch twin neural network;

[0043] If it is determined that the first characteristic distance is greater than a first preset distance threshold, calculating the second characteristic distances between two adjacent frame images and the normal sample image respectively based on the second branch twin neural network;

[0044] The first branch twin neural network and the second branch twin neural network are respectively trained according to corresponding contrast loss functions and using corresponding training data sets; the contrast loss function uses the square term of the Euclidean distance between features of samples as a factor to enhance the distance difference and smooth the optimization process;

[0045] If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target.

[0046] The embodiments of the present invention provide a method and device for detecting abnormal targets based on images taken by a drone, which obtain images taken by a drone and containing targets to be detected, and calculate the first characteristic distance between two adjacent frames of images based on a first-branch twin neural network; if it is determined that the first characteristic distance is greater than a first preset distance threshold, the second characteristic distances between the two adjacent frames of images and the normal sample image are calculated based on a second-branch twin neural network; if it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target, which can realize real-time, efficient and accurate detection of abnormal targets from drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0048] Figure 1 It is a flow chart of an abnormal target detection method based on images taken by a drone provided in one embodiment of the present invention.

[0049] Figure 2 It is a flow chart of an abnormal target detection method based on images taken by a drone provided by another embodiment of the present invention.

[0050] Figure 3 It is a flow chart of an abnormal target detection method based on images taken by a drone provided by another embodiment of the present invention.

[0051] Figure 4 It is a structural schematic diagram of an abnormal target detection device based on images taken by a drone provided by an embodiment of the present invention.

[0052] Figure 5 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily.

[0054] Figure 1FIG. 1 is a flow chart of an abnormal target detection method based on images taken by a drone provided by an embodiment of the present invention. Figure 1 As shown, the abnormal target detection method based on the image taken by the drone provided by the embodiment of the present invention includes:

[0055] Step S1: Obtain an image containing the target to be detected taken by the drone, and calculate the first feature distance between two adjacent frames of the image based on the first branch twin neural network.

[0056] Step S2: if it is determined that the first characteristic distance is greater than a first preset distance threshold, then calculating the second characteristic distances between two adjacent frame images and the normal sample image based on the second branch twin neural network;

[0057] Among them, the first-branch twin neural network and the second-branch twin neural network are respectively trained according to the corresponding contrast loss function and using the corresponding training data set; the contrast loss function uses the square term of the Euclidean distance of the features between samples as a factor to enhance the distance difference and smooth the optimization process.

[0058] Step S3: If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, then the target to be detected in the target frame image is determined to be an abnormal target.

[0059] In the above step S1, the device obtains an image taken by a drone containing a target to be detected, and calculates the first characteristic distance between two adjacent frames of the image based on the first branch of the twin neural network. The device may be a computer device that executes the method, for example, it may include a drone with a processor inside. It should be noted that the acquisition and analysis of data involved in the embodiments of the present invention are authorized by the user. The target to be detected may include a cable, and the purpose of anomaly detection is to determine whether the above cable is damaged. The Siamese Neural Network is an architecture that includes two sub-networks. The sub-networks have the same network structure, parameters and weights. The update of parameters is performed simultaneously on the two sub-networks. Sharing weights can reduce the number of training parameters and improve training efficiency.

[0060] The real-time video collected by the drone can be preprocessed by video frame segmentation to obtain each frame image containing the target to be detected, and the two adjacent frame images Pi-1 and Pi are input into the first branch twin neural network to calculate the first feature distance between the two adjacent frame images.

[0061] In the above step S2, if the device determines that the first characteristic distance is greater than the first preset distance threshold, the second characteristic distance between two adjacent frame images and the normal sample image is calculated based on the second branch twin neural network;

[0062] Among them, the first branch twin neural network and the second branch twin neural network are respectively trained according to the corresponding contrast loss function and the corresponding training data set; the contrast loss function uses the square term of the Euclidean distance between the features of the samples as a factor to enhance the distance difference and smooth the optimization process. The first preset distance threshold can be set independently according to the actual situation. If the first characteristic distance is greater than the first preset distance threshold, it means that there is a change node in the two adjacent frames of images. The two adjacent frames of images Pi-1 and Pi are input into the second branch twin neural network to calculate the second characteristic distance between the two adjacent frames of images and the normal sample image, that is, the second characteristic distance between Pi-1 and the normal sample image is calculated, and the second characteristic distance between Pi and the normal sample image is calculated. The changing nodes are explained as follows:

[0063] The change node of a video refers to the connection point between two adjacent pictures, also known as the editing point.

[0064] That is, the function of the first branch of the twin neural network is to determine whether there are changed nodes between two adjacent frames of images, and the function of the second branch of the twin neural network is target anomaly detection.

[0065] The abnormal target detection method based on the image taken by the drone also includes:

[0066] If it is determined that the first characteristic distance is less than or equal to the first preset distance threshold, the first characteristic distance between the two subsequent adjacent frame images is calculated based on the first branch twin neural network traversal. If the first characteristic distance is less than or equal to the first preset distance threshold, it means that there are unchanged nodes in the two adjacent frame images, and no operation is required, and then the first branch twin neural network is used to continue to judge the subsequent adjacent frame images.

[0067] In the above step S3, if the device determines that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target. The second preset distance threshold can be set independently according to actual conditions. If the second characteristic distance between Pi-1 and the normal sample image is greater than the second preset distance threshold, it means that the frame image Pi-1 contains an abnormal target. If the second characteristic distance between Pi and the normal sample image is greater than the second preset distance threshold, it means that the frame image Pi contains an abnormal target.

[0068] After determining that an abnormal target is included in the frame image, an alarm message can be further generated. The alarm message can record relevant information; the relevant information can include the alarm occurrence time, the current frame image, the UAV position, etc. At the same time, combined with manual feedback and actual situations, the detection models (the first-branch Siamese neural network and the second-branch Siamese neural network) are adjusted and optimized, including but not limited to adjusting the distance thresholds (the first preset distance threshold and the second preset distance threshold), updating the model parameters, introducing new training data, etc.

[0069] When the target to be detected just shows an abnormality, the frame image corresponding to that moment can be detected in real time through the first-branch Siamese neural network, and the abnormal target can be accurately detected through the second-branch Siamese neural network. After the method of the present invention completes the training of the first-branch Siamese neural network and the second-branch Siamese neural network, no manual intervention is required, and the detection of abnormal targets can be efficiently achieved.

[0070] The abnormal target detection method based on the UAV-captured images further includes:

[0071] Parallelly execute calculating the first feature distance between two adjacent frame images based on the first-branch Siamese neural network, and calculating the second feature distance between each of the two adjacent frame images and the normal sample image based on the second-branch Siamese neural network. Figure 2 What is reflected is the serial execution of calculating the first feature distance between two adjacent frame images based on the first-branch Siamese neural network, and calculating the second feature distance between each of the two adjacent frame images and the normal sample image based on the second-branch Siamese neural network. The parallel execution situation can be described as follows:

[0072] For example, the first-branch Siamese neural network is calculating the first feature distance between two adjacent frame images Pi-1 and Pi, and the second-branch Siamese neural network is calculating the second feature distance between two adjacent frame images Pj-1 and Pj and the normal sample image respectively, where j < i, indicating that the two adjacent frame images calculated by the second-branch Siamese neural network occur before the two adjacent frame images calculated by the first-branch Siamese neural network. In this way, it can be ensured that the two-branch Siamese neural networks independently process their respective calculation tasks and reduce the waiting time. The synchronization of the same two adjacent frame images in the two-branch Siamese neural networks can be achieved through the frame numbers (i or j) of the two adjacent frame images.

[0073] Pre-train the first-branch Siamese neural network and the second-branch Siamese neural network, including:

[0074] Obtain the historical video data captured by the UAV, and the historical video data contains the target to be detected;

[0075] A training data set is constructed according to the historical video data, wherein the training data set includes a first training data set consisting of continuous frame similar samples and continuous frame dissimilar samples, and a second training data set consisting of normal samples and abnormal samples; Figure 3 As shown, the historical video data can be processed by frame division, cleaning, enhancement and standardization, and the processed historical video data can be used to add labels to construct a training data set. For the data in the first training data set, the corresponding labels are the labels of the continuous frame sample pairs (similar is 0, dissimilar is 1); for the data in the second training data set, the corresponding labels are the labels of the normal sample and abnormal sample pairs (normal is 0, abnormal is 1). The above data enhancement can be achieved by rotation, scaling, cropping, flipping, etc.

[0076] Construct the first-branch twin neural network and the second-branch twin neural network, and use the first training data set to train the first-branch twin neural network until the first-branch twin neural network converges, and use the second training data set to train the second-branch twin neural network until the second-branch twin neural network converges.

[0077] The constructing of the first branch twin neural network and the second branch twin neural network comprises:

[0078] A detail feature extraction layer is constructed based on a deconvolutional neural network, and a global feature capture layer is constructed based on the TransformerEncoder of a single-head attention mechanism. The detail feature extraction layer and the global feature capture layer are connected in series to obtain the first branch twin neural network and the second branch twin neural network. That is, the first branch twin neural network and the second branch twin neural network have the same network structure and both have the ability to extract detail features and global features. The detail feature extraction layer is used to extract detail features of the input image through a deconvolutional neural network; the global feature capture layer is used to input the detail features into the Transformer Encoder to capture the spatial correlation features between image pixels. The spatial correlation features between pixels are described as follows:

[0079] The spatial correlation features between pixels mainly include neighborhood, adjacency, connection, connectivity and distance.

[0080] The neighborhood of a pixel refers to the set of adjacent pixels of a pixel.

[0081] Pixel adjacency refers to the contact relationship between a pixel and the pixels in its neighborhood.

[0082] Connectivity refers to the relationship between two pixels, which needs to meet two conditions: first, whether the two pixels are adjacent in space; second, whether their grayscale values ​​meet a certain similarity criterion.

[0083] Connectivity refers to a channel consisting of a series of sequentially connected pixels that can connect from one point to another. Connectivity requires two conditions to be met: first, whether these pixels are adjacent image subsets; second, whether the grayscale values ​​of adjacent pixels in the image subset meet a certain similarity criterion.

[0084] The distance between pixels is a quantitative indicator to measure the positional relationship between two pixels. Common distance measurement functions include Euclidean distance, urban area distance, and chessboard distance.

[0085] The step of training the first branch twin neural network using the first training data set includes:

[0086] The first branch Siamese neural network is trained using the first training data set according to the following first contrast loss function:

[0087]

[0088] Wherein, L1 is the first contrast loss function, Y1 is the label of the continuous frame sample pair, , in order to balance the weights between similar samples and dissimilar samples, m1 is the first preset distance threshold, and D1 is the Euclidean distance of features between similar samples and dissimilar samples. The purpose of using the first contrast loss function for model training is to make the feature distance of similar continuous frame samples as small as possible and the feature distance of dissimilar continuous frame samples as large as possible.

[0089] Square term of D1: The present invention uses the square term of D1 (the square of the Euclidean distance between the features of normal samples and abnormal samples) instead of directly using D1, which has the following technical effects:

[0090] Enhanced distance difference: The square term amplifies the distance difference, making the model pay more attention to the distance changes between samples. This helps the model more effectively narrow the distance between normal samples during training, while pushing the distance between normal and abnormal samples.

[0091] Smooth optimization process: The square term makes the loss function smoother, which helps the optimization algorithm converge faster and avoid falling into local optimal solutions.

[0092] Weight α1: The weight α1 is introduced to balance the loss contribution between normal samples and abnormal samples, which has the following technical effects:

[0093] Solve the problem of class imbalance: In many anomaly detection tasks, the number of normal samples is far greater than that of abnormal samples. By adjusting α1, we can ensure that the model pays more attention to abnormal samples during training, thereby improving the accuracy of anomaly detection.

[0094] Flexible adjustment: The value of α1 can be adjusted according to specific application scenarios to adapt it to different data distributions and task requirements.

[0095] max(): max() is used to ensure that when the distance between normal samples and abnormal samples is less than the preset threshold m1, the model will be penalized, which has the following technical effects:

[0096] Clear boundaries: By setting the threshold m1, the boundary between normal samples and abnormal samples is clarified, so that the model can more clearly divide the two types of samples in the feature space.

[0097] Reducing false positives: This mechanism reduces the false positive rate due to feature space overlap and improves the robustness and reliability of the model.

[0098] The step of training the second branch twin neural network using the second training data set includes:

[0099] The second branch Siamese neural network is trained using the second training data set according to the following second contrast loss function:

[0100]

[0101] Among them, L2 is the second contrast loss function, Y2 is the label of the normal sample and the abnormal sample pair, , in order to balance the weight between normal samples and abnormal samples, m2 is the second preset distance threshold, and D2 is the Euclidean distance of the features between normal samples and abnormal samples. The purpose of using the second contrast loss function for model training is to make the feature distance between normal samples as small as possible and the feature distance between normal samples and abnormal samples as large as possible.

[0102] Square term of D2: The present invention uses the square term of D2 (the square of the Euclidean distance between the features of normal samples and abnormal samples) instead of directly using D2, which has the following technical effects:

[0103] Enhanced distance difference: The square term amplifies the distance difference, making the model pay more attention to the distance changes between samples. This helps the model more effectively narrow the distance between normal samples during training, while pushing the distance between normal and abnormal samples.

[0104] Smooth optimization process: The square term makes the loss function smoother, which helps the optimization algorithm converge faster and avoid falling into local optimal solutions.

[0105] Weight α2: The weight α2 is introduced to balance the loss contribution between normal samples and abnormal samples, which has the following beneficial technical effects:

[0106] Solve the problem of class imbalance: In many anomaly detection tasks, the number of normal samples is far greater than that of abnormal samples. By adjusting α2, we can ensure that the model pays more attention to abnormal samples during training, thereby improving the accuracy of anomaly detection.

[0107] Flexible adjustment: The value of α2 can be adjusted according to specific application scenarios to adapt it to different data distributions and task requirements.

[0108] max(): max() is used to ensure that when the distance between normal samples and abnormal samples is less than the preset threshold m2, the model will be penalized, which has the following technical effects:

[0109] Clear boundaries: By setting the threshold m2, the boundary between normal samples and abnormal samples is clarified, so that the model can more clearly divide the two types of samples in the feature space.

[0110] Reducing false positives: This mechanism reduces the false positive rate due to feature space overlap and improves the robustness and reliability of the model.

[0111] The abnormal target detection method based on drone-photographed images provided in the embodiment of the present invention can be implemented based on modularization, including a data collection module, a data processing module, a model building module, an abnormality detection module, an alarm module and a feedback optimization module; wherein:

[0112] Data collection module: used to obtain historical video data of the drone and upload the data to the data processing module;

[0113] Data processing module: used to frame and pre-process the data according to the collected information, construct the first and second training data sets, and upload the data sets to the model construction module;

[0114] Model building module: used to build a multi-branch twin neural network abnormal target detection model based on the received data information and the deconvolution and single-head self-attention Transform Encoder modules;

[0115] Anomaly detection module: used to detect abnormal targets in real-time drone video using a multi-branch twin neural network abnormal target detection model;

[0116] Alarm module: records the alarm information when abnormal targets appear and sends alarms to staff;

[0117] Feedback optimization module: used to adjust and optimize the multi-branch twin neural network abnormal target detection model based on manual feedback and actual conditions.

[0118] The abnormal target detection method based on drone-photographed images provided by the embodiment of the present invention has the following beneficial technical effects:

[0119] The present invention constructs an image feature extraction network based on deconvolution and single-head self-attention Transform Encoder modules to deeply extract detail features in the image and spatial correlation features between pixels; first, the first branch twin neural network is used to identify the change nodes in the video, quickly detect possible anomalies or important changes, and then the second branch twin neural network is used to perform a deeper analysis and confirmation based on the first stage of anomaly detection, compare the feature distances between adjacent frames of the change nodes and normal samples, and further verify and confirm the existence of abnormal targets. This method can quickly and accurately detect abnormal targets in drone videos, effectively reduce the false alarm rate, and improve the accuracy and reliability of anomaly detection.

[0120] The abnormal target detection method based on drone-photographed images provided in an embodiment of the present invention obtains an image taken by a drone containing a target to be detected, and calculates a first characteristic distance between two adjacent frames of images based on a first-branch twin neural network; if it is determined that the first characteristic distance is greater than a first preset distance threshold, then the second characteristic distances between the two adjacent frames of images and the normal sample image are calculated based on a second-branch twin neural network; if it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, then the target to be detected in the target frame image is determined to be an abnormal target, which can realize real-time, efficient and accurate drone abnormal target detection.

[0121] Furthermore, the abnormal target detection method based on drone-photographed images also includes:

[0122] If it is determined that the first characteristic distance is less than or equal to the first preset distance threshold, the first characteristic distance between two subsequent adjacent frames of images is calculated based on the first branch twin neural network traversal. The above embodiment can be referred to for description and will not be repeated here.

[0123] Furthermore, the abnormal target detection method based on drone-photographed images also includes:

[0124] The first characteristic distance between two adjacent frames of images is calculated based on the first branch of the twin neural network, and the second characteristic distance between two adjacent frames of images and the normal sample image is calculated based on the second branch of the twin neural network in parallel. Please refer to the above embodiment for description, and no further details will be given.

[0125] Further, pre-training the first branch twin neural network and the second branch twin neural network includes:

[0126] The historical video data shot by the drone is obtained, and the historical video data contains the target to be detected; the description can be made with reference to the above embodiment and will not be repeated here.

[0127] A training data set is constructed based on the historical video data, and the training data set includes a first training data set consisting of continuous frame similar samples and continuous frame dissimilar samples, and a second training data set consisting of normal samples and abnormal samples; the above-mentioned embodiment can be referred to for description and will not be repeated here.

[0128] The first branch twin neural network and the second branch twin neural network are constructed, and the first branch twin neural network is trained using the first training data set until the first branch twin neural network converges, and the second branch twin neural network is trained using the second training data set until the second branch twin neural network converges. The above-mentioned embodiments can be referred to for explanation and will not be described in detail.

[0129] Furthermore, the step of constructing the first branch twin neural network and the second branch twin neural network includes:

[0130] A detail feature extraction layer is constructed based on a deconvolutional neural network, and a global feature capture layer is constructed based on a TransformerEncoder with a single-head attention mechanism. The detail feature extraction layer and the global feature capture layer are connected in series to obtain the first branch twin neural network and the second branch twin neural network. Please refer to the above embodiment for description and do not repeat them here.

[0131] Further, the using the first training data set to train the first branch twin neural network includes:

[0132] The first branch Siamese neural network is trained using the first training data set according to the following first contrast loss function:

[0133]

[0134] Wherein, L1 is the first contrast loss function, Y1 is the label of the continuous frame sample pair, , is to balance the weights between similar samples and dissimilar samples, m1 is the first preset distance threshold, and D1 is the Euclidean distance of features between similar samples and dissimilar samples. The above embodiments can be referred to for explanation and will not be repeated here.

[0135] Further, the using the second training data set to train the second branch twin neural network includes:

[0136] The second branch Siamese neural network is trained using the second training data set according to the following second contrast loss function:

[0137]

[0138] Among them, L2 is the second contrast loss function, Y2 is the label of the normal sample and the abnormal sample pair, , is to balance the weight between normal samples and abnormal samples, m2 is the second preset distance threshold, and D2 is the Euclidean distance of features between normal samples and abnormal samples. The above embodiments can be referred to for explanation and will not be repeated here.

[0139] Figure 4 FIG. 1 is a schematic diagram of a structure of an abnormal target detection device based on images taken by a drone according to an embodiment of the present invention. Figure 4 As shown, the abnormal target detection device based on the image taken by the drone provided in the embodiment of the present invention includes an acquisition unit 401, a calculation unit 402 and a detection unit 403, wherein:

[0140] The acquisition unit 401 is used to acquire an image taken by a drone containing a target to be detected, and calculate the first characteristic distance between two adjacent frames of images based on the first branch twin neural network; the calculation unit 402 is used to calculate the second characteristic distance between two adjacent frames of images and the normal sample image based on the second branch twin neural network if it is determined that the first characteristic distance is greater than the first preset distance threshold; wherein the first branch twin neural network and the second branch twin neural network are respectively trained according to the corresponding contrast loss function and using the corresponding training data set; the contrast loss function uses the square term of the Euclidean distance of the features between samples as a factor to enhance the distance difference and the smooth optimization process; the detection unit 403 is used to determine that the target to be detected in the target frame image is an abnormal target if it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold.

[0141] Specifically, the acquisition unit 401 in the device is used to acquire an image taken by a drone containing a target to be detected, and calculate the first characteristic distance between two adjacent frames of images based on the first branch twin neural network; the calculation unit 402 is used to calculate the second characteristic distances between two adjacent frames of images and the normal sample image based on the second branch twin neural network if it is determined that the first characteristic distance is greater than the first preset distance threshold; wherein the first branch twin neural network and the second branch twin neural network are respectively trained according to the corresponding contrast loss function and using the corresponding training data set; the contrast loss function uses the square term of the Euclidean distance of the features between samples as a factor to enhance the distance difference and the smooth optimization process; the detection unit 403 is used to determine that the target to be detected in the target frame image is an abnormal target if it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold.

[0142] The abnormal target detection device based on drone-photographed images provided by an embodiment of the present invention obtains an image taken by a drone containing a target to be detected, and calculates a first characteristic distance between two adjacent frames of images based on a first-branch twin neural network; if it is determined that the first characteristic distance is greater than a first preset distance threshold, then the second characteristic distance between the two adjacent frames of images and the normal sample image is calculated based on a second-branch twin neural network; if it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, then the target to be detected in the target frame image is determined to be an abnormal target, which can realize real-time, efficient and accurate drone abnormal target detection.

[0143] The embodiment of the present invention provides an embodiment of an abnormal target detection device based on images taken by a drone, which can be specifically used to execute the processing flow of the above-mentioned method embodiments. Its functions are not repeated here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0144] Figure 5 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 5 As shown, the electronic device includes: a processor (processor) 501, a memory (memory) 502 and a bus 503;

[0145] The processor 501 and the memory 502 communicate with each other via a bus 503;

[0146] The processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above method embodiments, for example, including:

[0147] Obtain an image containing a target to be detected taken by a drone, and calculate a first feature distance between two adjacent frames of the image based on a first-branch twin neural network;

[0148] If it is determined that the first characteristic distance is greater than a first preset distance threshold, calculating the second characteristic distances between two adjacent frame images and the normal sample image respectively based on the second branch twin neural network;

[0149] The first branch twin neural network and the second branch twin neural network are respectively trained according to corresponding contrast loss functions and using corresponding training data sets; the contrast loss function uses the square term of the Euclidean distance between features of samples as a factor to enhance the distance difference and smooth the optimization process;

[0150] If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target.

[0151] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the methods provided by the above method embodiments, for example, including:

[0152] Obtain an image containing a target to be detected taken by a drone, and calculate a first feature distance between two adjacent frames of the image based on a first-branch twin neural network;

[0153] If it is determined that the first characteristic distance is greater than a first preset distance threshold, calculating the second characteristic distances between two adjacent frame images and the normal sample image respectively based on the second branch twin neural network;

[0154] The first branch twin neural network and the second branch twin neural network are respectively trained according to corresponding contrast loss functions and using corresponding training data sets; the contrast loss function uses the square term of the Euclidean distance between features of samples as a factor to enhance the distance difference and smooth the optimization process;

[0155] If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target.

[0156] This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program enables the computer to execute the methods provided by the above method embodiments, for example, including:

[0157] Obtain an image containing a target to be detected taken by a drone, and calculate a first feature distance between two adjacent frames of the image based on a first-branch twin neural network;

[0158] If it is determined that the first characteristic distance is greater than a first preset distance threshold, calculating the second characteristic distances between two adjacent frame images and the normal sample image respectively based on the second branch twin neural network;

[0159] The first branch twin neural network and the second branch twin neural network are respectively trained according to corresponding contrast loss functions and using corresponding training data sets; the contrast loss function uses the square term of the Euclidean distance between features of samples as a factor to enhance the distance difference and smooth the optimization process;

[0160] If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target.

[0161] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] In the description of this specification, the description with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0166] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting abnormal targets based on images taken by drones, characterized in that: include: Obtain an image containing a target to be detected taken by a drone, and calculate a first feature distance between two adjacent frames of the image based on a first-branch twin neural network; If it is determined that the first characteristic distance is greater than a first preset distance threshold, calculating the second characteristic distances between two adjacent frame images and the normal sample image respectively based on the second branch twin neural network; Among them, the first branch twin neural network and the second branch twin neural network are respectively trained according to the corresponding contrast loss function and using the corresponding training data set; the contrast loss function uses the square term of the Euclidean distance of the features between samples as a factor to enhance the distance difference and smooth the optimization process; for the first branch twin neural network, the features between samples are the features between similar samples and dissimilar samples; for the second branch twin neural network, the features between samples are the features between normal samples and abnormal samples; If it is determined that there is at least one target frame image whose second characteristic distance is greater than the second preset distance threshold, the target to be detected in the target frame image is determined to be an abnormal target.

2. The abnormal target detection method based on drone-photographed images according to claim 1 is characterized in that: The abnormal target detection method based on the image taken by the drone also includes: If it is determined that the first feature distance is less than or equal to the first preset distance threshold, continue to calculate the first feature distance between the subsequent two adjacent frame images based on the first branch twin neural network traversal.

3. The abnormal target detection method based on drone-photographed images according to claim 1 is characterized in that: The abnormal target detection method based on the image taken by the drone also includes: The first characteristic distance between two adjacent frames of images is calculated based on the first branch of the twin neural network, and the second characteristic distance between the two adjacent frames of images and the normal sample image is calculated based on the second branch of the twin neural network in parallel.

4. The abnormal target detection method based on drone-photographed images according to any one of claims 1 to 3, characterized in that: Pre-training the first branch twin neural network and the second branch twin neural network includes: Acquire historical video data shot by a drone, wherein the historical video data contains the target to be detected; Constructing a training data set according to the historical video data, wherein the training data set includes a first training data set consisting of continuous frame similar samples and continuous frame dissimilar samples, and a second training data set consisting of normal samples and abnormal samples; Construct the first-branch twin neural network and the second-branch twin neural network, and use the first training data set to train the first-branch twin neural network until the first-branch twin neural network converges, and use the second training data set to train the second-branch twin neural network until the second-branch twin neural network converges.

5. The abnormal target detection method based on drone-photographed images according to claim 4 is characterized in that: The constructing of the first branch twin neural network and the second branch twin neural network comprises: A detail feature extraction layer is constructed based on a deconvolutional neural network, and a global feature capture layer is constructed based on a TransformerEncoder with a single-head attention mechanism. The detail feature extraction layer and the global feature capture layer are connected in series to obtain the first-branch twin neural network and the second-branch twin neural network.

6. The abnormal target detection method based on drone-photographed images according to claim 4 is characterized in that: The step of training the first branch twin neural network using the first training data set includes: The first branch Siamese neural network is trained using the first training data set according to the following first contrast loss function: ； Wherein, L1 is the first contrast loss function, Y1 is the label of the continuous frame sample pair, , is to balance the weights between similar samples and dissimilar samples, m1 is the first preset distance threshold, and D1 is the Euclidean distance of features between similar samples and dissimilar samples.

7. The abnormal target detection method based on drone-photographed images according to claim 4 is characterized in that: The step of training the second branch twin neural network using the second training data set includes: The second branch Siamese neural network is trained using the second training data set according to the following second contrast loss function: ; Among them, L2 is the second contrast loss function, Y2 is the label of the normal sample and the abnormal sample pair, , is to balance the weight between normal samples and abnormal samples, m2 is the second preset distance threshold, and D2 is the Euclidean distance of the features between normal samples and abnormal samples.

8. An abnormal target detection device based on images taken by drones, characterized in that: include: An acquisition unit, used for acquiring an image containing a target to be detected taken by a drone, and calculating a first characteristic distance between two adjacent frames of images based on a first branch twin neural network; A calculation unit, configured to calculate, based on a second branch twin neural network, second characteristic distances between two adjacent frame images and a normal sample image if it is determined that the first characteristic distance is greater than a first preset distance threshold; Among them, the first branch twin neural network and the second branch twin neural network are respectively trained according to the corresponding contrast loss function and using the corresponding training data set; the contrast loss function uses the square term of the Euclidean distance of the features between samples as a factor to enhance the distance difference and smooth the optimization process; for the first branch twin neural network, the features between samples are the features between similar samples and dissimilar samples; for the second branch twin neural network, the features between samples are the features between normal samples and abnormal samples; The detection unit is used to determine that the target to be detected in the target frame image is an abnormal target if it is determined that there is at least one target frame image whose second characteristic distance is greater than a second preset distance threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images