Dual-view fusion unmanned aerial vehicle detection method based on event camera detection

Abstract

The invention discloses a dual-view fusion unmanned aerial vehicle detection method based on event camera detection, and the method comprises the steps: obtaining an original event point cloud of a target scene through an event camera; extracting point cloud features and voxel features of the original event point cloud; fusing the point cloud features and the voxel features to obtain target fusion features of the original event point cloud; and detecting according to the target fusion feature to obtain an unmanned aerial vehicle detection result of the target scene, wherein the unmanned aerial vehicle detection result is used for indicating the unmanned aerial vehicle in the target scene.

Description

Technical Field

[0001] The present application relates to the field of drone detection, and in particular to a dual-view fusion drone detection method based on event camera detection. Background Art

[0002] With the development of low-altitude airspace, drones are gradually being used in various industries, such as remote monitoring, precision agriculture, geological exploration, meteorological observation, power inspection, emergency rescue, express logistics, and film and television shooting. However, improper use of drones often causes a variety of safety hazards and social problems. Therefore, how to detect and track drone activities in low-altitude airspace in real time and accurately has become an urgent problem to be solved.

[0003] Traditional detection technologies, such as acoustic detection, radio detection, and radar detection, have high detection accuracy and range, but are often accompanied by high equipment costs and complex configuration requirements, which limit their application in certain scenarios. In contrast, machine vision-based detection technology has gradually become a research hotspot in the field of low-altitude drone detection due to its low cost and easy configuration.

[0004] Traditional frame-based cameras have limited performance under extreme lighting conditions (such as overexposure or dark scenes). This limitation directly affects the effective capture and identification of drone targets. In order to overcome this problem, event cameras, as a new type of neuromorphic sensor, provide new ideas for the development of low-altitude drone detection technology with their unique advantages. The dynamic vision sensor (DVS) based on the address-event representation (AER) mechanism of event cameras provides an efficient method for processing high-speed, high dynamic range visual information. In AER, each event point contains key information: the pixel location of the event, the timestamp, and the event polarity, which are very effective for capturing and analyzing fast-moving or changing targets.

[0005] Some existing event camera-based object detection methods directly use single event representations generated from short time windows of events to infer detections. These methods discard relevant information outside the considered time window, making it difficult or impossible to detect slow-moving objects. Other methods address this problem by introducing recurrent neural network layers, which greatly improves detection performance. However, when applied to drone detection, due to the long imaging distance and the small size of drones on the image plane, they appear as shapeless and intensityless continuously moving pixels in the event stream data, resulting in the inability of the above existing algorithms to effectively detect targets. Summary of the invention

[0006] To this end, this application discloses the following technical solutions:

[0007] The first aspect of the present application provides a dual-view fusion drone detection method based on event camera detection, comprising:

[0008] Obtain the original event point cloud of the target scene through the event camera;

[0009] Extracting point cloud features and voxel features of the original event point cloud;

[0010] Fusing the point cloud features and the voxel features to obtain target fusion features of the original event point cloud;

[0011] A drone detection result of the target scene is obtained according to the target fusion feature detection, and the drone detection result is used to indicate a drone in the target scene.

[0012] Optionally, the fusing the point cloud feature and the voxel feature to obtain a target fusion feature of the original event point cloud includes:

[0013] Performing convolution processing and normalization processing on a spliced ​​feature formed by splicing the point cloud feature and the voxel feature to obtain a fusion probability weight;

[0014] The point cloud features and the voxel features are weightedly summed according to the fusion probability weights to obtain target fusion features of the original event point cloud.

[0015] Optionally, performing weighted summation on the point cloud features and the voxel features according to the fusion probability weight to obtain the target fusion features of the original event point cloud includes:

[0016] Separating the point cloud channel and the voxel channel of the fusion probability weight;

[0017] The product of the point cloud channel and the point cloud feature and the product of the voxel channel and the voxel feature are summed to obtain the target fusion feature of the original event point cloud.

[0018] Optionally, extracting point cloud features and voxel features of the original event point cloud includes:

[0019] Processing the original event point cloud using the point cloud feature branch of the dual-branch segmentation network to obtain point cloud features;

[0020] The original event point cloud is discretized to obtain point cloud voxels, and the point cloud voxels are processed using a voxel feature branch of a dual-branch segmentation network to obtain voxel features.

[0021] Optionally, the point cloud feature branch includes a first multi-layer perceptron, a second multi-layer perceptron, a third multi-layer perceptron and a fourth multi-layer perceptron;

[0022] The voxel feature branch includes an input layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a fourth down-sampling layer, a first up-sampling layer, a second up-sampling layer, a third up-sampling layer and a fourth up-sampling layer.

[0023] Optionally, the step of processing the original event point cloud using a point cloud feature branch of a dual-branch segmentation network to obtain point cloud features includes:

[0024] Processing the original event point cloud using the first multi-layer perceptron to obtain a first point cloud feature;

[0025] Processing the first fusion feature using the second multi-layer perceptron to obtain a second point cloud feature;

[0026] Processing the second fusion feature using the third multi-layer perceptron to obtain a third point cloud feature;

[0027] Processing the third fusion feature using the fourth multi-layer perceptron to obtain a fourth point cloud feature;

[0028] The step of processing the point cloud voxels using the voxel feature branch of the dual-branch segmentation network to obtain voxel features includes:

[0029] Processing the point cloud voxels using the input layer to obtain first voxel features;

[0030] Processing the first fusion feature in sequence by using the first downsampling layer, the second downsampling layer, the third downsampling layer and the fourth downsampling layer to obtain a second voxel feature;

[0031] Using the first upsampling layer and the second upsampling layer to sequentially process the second fusion feature to obtain a third voxel feature;

[0032] Using the third upsampling layer and the fourth upsampling layer to sequentially process the third fusion feature to obtain a fourth voxel feature;

[0033] The first fusion feature is obtained by fusing the first point cloud feature and the first voxel feature;

[0034] The second fusion feature is obtained by fusing the second point cloud feature and the second voxel feature;

[0035] The third fusion feature is obtained by fusing the third point cloud feature and the third voxel feature;

[0036] The fourth point cloud feature and the fourth voxel feature are used to fuse to obtain the target fusion feature.

[0037] Optionally, evaluation indicators of the dual-branch segmentation network include overlap, average accuracy, trajectory detection rate and false alarm rate.

[0038] Optionally, the method for training the dual-branch segmentation network includes:

[0039] Processing the sample event point cloud according to the dual-branch segmentation network to obtain a drone detection result of the sample event point cloud;

[0040] Determining the network loss of the dual-branch segmentation network according to the drone detection result of the sample event point cloud and the preset drone labeling result of the sample event point cloud;

[0041] The network parameters of the dual-branch segmentation network are updated according to the network loss until the network loss meets a preset convergence condition.

[0042] Optionally, the convergence condition includes:

[0043] The network loss obtained this time is less than a preset first threshold;

[0044] or,

[0045] The difference between the network loss obtained this time and the network loss obtained last time is less than a preset second threshold.

[0046] Optionally, extracting point cloud features and voxel features of the original event point cloud includes:

[0047] Dividing the original event point cloud into a plurality of basic processing units using a time window with a fixed time interval;

[0048] Extracting point cloud features and voxel features of at least one of the basic processing units.

[0049] The beneficial effects of this application are:

[0050] In order to solve the problems faced by event cameras in low-altitude UAV detection, such as small target size, few features, sparse and irregular events, this application proposes a solution that fuses the point cloud features and voxel features of the original event point cloud to detect UAV targets in the original event point cloud. Compared with the current most advanced target detection and point cloud segmentation networks, this solution can have the highest detection performance while maintaining low false alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 is a flow chart of a dual-view fusion drone detection method based on event camera detection provided by an embodiment of the present application;

[0053] Figure 2 It is a structural diagram of a fusion module provided in an embodiment of the present application;

[0054] Figure 3 It is a structural diagram of a dual-branch segmentation network provided in an embodiment of the present application;

[0055] Figure 4 It is a schematic diagram of a YOLOV7 detection result and a real scene provided in an embodiment of the present application;

[0056] Figure 5 is a schematic diagram of a detection result of Pointnet++ provided in an embodiment of the present application and a detection result of the method of this embodiment;

[0057] Figure 6 It is a schematic diagram of a correct detection result provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] This application embodiment provides a dual-view fusion drone detection method based on event camera detection, see Figure 1 , is a flow chart of the method, and the method may include the following steps.

[0060] S101, obtaining an original event point cloud of a target scene through an event camera.

[0061] S102, extracting point cloud features and voxel features of the original event point cloud.

[0062] S103, fusing the point cloud features and the voxel features to obtain target fusion features of the original event point cloud.

[0063] S104, obtaining a drone detection result of the target scene according to the target fusion feature detection, and the drone detection result is used to indicate the drone in the target scene.

[0064] The beneficial effects of this application are:

[0065] In order to solve the problems faced by event cameras in low-altitude UAV detection, such as small target size, few features, sparse and irregular events, this application proposes a solution that fuses the point cloud features and voxel features of the original event point cloud to detect UAV targets in the original event point cloud. Compared with the current most advanced target detection and point cloud segmentation networks, this solution can have the highest detection performance while maintaining low false alarms.

[0066] In step S101, an event camera can be used to shoot the target scene that needs to be detected by the drone to obtain event stream data output by the event camera. For example, an event camera can be used to shoot low-altitude airspace at a specific direction and height to obtain corresponding event stream data.

[0067] The event stream data output by the event camera can be regarded as a point cloud composed of two-dimensional spatial position information and time information. Compared with the laser radar point cloud, this special point cloud composed of event points does not contain the depth information of the target, but the time information constitutes its third dimension. In S101, the event stream data output by the event camera can be used as the above-mentioned original event point cloud.

[0068] In this embodiment, the original event point cloud can be divided into n basic processing units ε by using a time window with a fixed time interval. i , the value range of i is 1 to n, that is, E = (ε1, ε2, ... ε n ), where each basic processing unit corresponds to a time window, and each basic processing unit includes N asynchronous binary discrete points e timestamps within the corresponding time window. k , that is, ε i ={e k =(x k ,y k , t k , p k )|k=1~N}, in this embodiment, each asynchronous binary discrete point e k It can be recorded as an event point.

[0069] where x k and k Represents the position of the discrete point in the spatial dimension, x k The value range is from 1 to W, y k The value range is from 1 to H, W and H represent the width and height range of the event stream data, t kIndicates the timestamp, p k The polarity value that indicates the brightness change. Its specific value can be 0 or 1. A value equal to 0 indicates that the pixel brightness decreases, and a value equal to 1 indicates that the pixel brightness increases.

[0070] In steps S102 to S104, multiple event points included in any one of the basic processing units may be processed to obtain the drone detection result of the target scene within the corresponding time window, or all the basic processing units may be processed to obtain the drone detection result of the target scene within the entire time period covered by the event stream data. In other words, the original event point cloud used to extract features in step S102 may be a set E of the above multiple basic processing units, or may be any one of the above basic processing units ε. i It can be understood that, no matter which case, the original event point cloud is composed of multiple event points.

[0071] That is, when extracting the point cloud features and voxel features of the original event point cloud, the method of this embodiment can:

[0072] The original event point cloud is divided into multiple basic processing units using a time window with a fixed time interval;

[0073] Extracting point cloud features and voxel features of at least one basic processing unit.

[0074] Specifically, the point cloud features and voxel features of any one of the basic processing units can be extracted to obtain the drone detection results within the time window corresponding to the basic processing unit; the point cloud features and voxel features of all the basic processing units can also be extracted to obtain the drone detection results of the target scene within the entire time period covered by the event stream data.

[0075] Optionally, step S102 may be implemented by a pre-trained dual-branch segmentation network. Specifically, extracting point cloud features and voxel features of the original event point cloud may include:

[0076] The point cloud feature branch of the dual-branch segmentation network is used to process the original event point cloud to obtain point cloud features;

[0077] The original event point cloud is discretized to obtain point cloud voxels, and the voxel feature branch of the dual-branch segmentation network is used to process the point cloud voxels to obtain voxel features.

[0078] In order to obtain voxel features, each event point contained in the original event point cloud can be discretized first to obtain the voxel unit corresponding to each event point. In this embodiment, the resolution of the voxel unit can be 2 pixels × 2 pixels × 10 milliseconds. In other optional embodiments, the voxel unit can also have other resolutions. This resolution is only used as an example.

[0079] For any event point e i After processing the event point with the point cloud feature branch, the feature Xp of the event point can be obtained i , after processing the event point with the voxel feature branch (specifically, the voxel unit corresponding to the event point can be processed), the feature Xv of the event point can be obtained i , both are vectors in the real field of d dimensions, and the dimension d is determined by the structure of the two-branch segmentation network.

[0080] Xv i According to the event point e i The position of the voxel unit is determined, that is, Xv i =V(x, y, t).

[0081] The original event point cloud includes multiple event points. The set of features of all event points processed by the point cloud feature branch is the point cloud feature of the original event point cloud, denoted as Xp = {Xp i |i=1~S}, S is the total number of event points contained in the original event point cloud. Similarly, the set of features of all event points processed by the voxel feature branch is the voxel feature of the original event point cloud, denoted as Xv={Xv i |i=1~S}.

[0082] In the relevant technical field, the core task of dual-branch feature fusion is to aggregate useful information under the interference of a large amount of useless information. The commonly used operations for aggregating this information are addition and cascading, but both the addition operation and the cascading operation are affected by many non-information features. The cascading operation can be expressed by formula (1), and the addition operation can be expressed by formula (2).

[0083] X con =concat(X1,...,X L ) (1)

[0084]

[0085] X con represents the fused feature vector obtained through the cascade operation, X add Represents the fused feature vector obtained by the addition operation. Due to the diversity of features of each event point in the point cloud, the features of different view branches are not consistent in importance. However, the addition and concatenation feature fusion strategies ignore the usefulness of each feature vector and combine a large number of useless features with useful features during the fusion process.

[0086] To address this problem, this embodiment introduces a gating mechanism when fusing point cloud features and fusion features. The gating mechanism can adaptively aggregate information by measuring the importance of each feature.

[0087] Based on the gating mechanism, the method of fusing the point cloud features and the voxel features in step S103 to obtain the target fusion features of the original event point cloud may be:

[0088] The spliced ​​features formed by splicing the point cloud features and the voxel features are convoluted and normalized to obtain the fusion probability weight;

[0089] The point cloud features and voxel features are weighted and summed according to the fusion probability weight to obtain the target fusion features of the original event point cloud.

[0090] The above fusion process can be expressed by the following formula (3) and formula (4).

[0091] D=softmax(conv(concat(Xp,Xv))) (3)

[0092] X door =D·Xp+(1-D)·Xv(4)

[0093] Among them, D represents the fusion probability weight, X door It represents the target fusion feature of the original event point cloud obtained by weighted summation. Softmax represents normalization processing, conv represents convolution processing, and concat represents concatenating the point cloud feature Xp and the voxel feature Xv to form a concatenated feature.

[0094] The above process of fusing point cloud features and voxel features to obtain target fusion features can be implemented by a pre-built fusion module, which can have Figure 2 The fusion module can first concatenate the features of the two views to obtain the concatenated features, and then perform convolution and normalization on the concatenated features to obtain the fusion probability weight D, and then convert the part of the fusion probability weight D related to the voxel features (i.e. Figure 2 The voxel channel shown in the figure is separated and the voxel feature is multiplied by the voxel channel of the fusion probability weight D. At the same time, the part of the fusion probability weight D related to the point cloud feature (i.e. Figure 2 The point cloud channel shown in FIG. 1 is separated out, the point cloud channel with the fusion probability weight D is multiplied with the point cloud feature, and finally the product of the two is added to obtain the final target fusion feature.

[0095] That is, the process of weighted summing point cloud features and voxel features according to the fusion probability weight to obtain the target fusion features of the original event point cloud may include:

[0096] Separate the point cloud channel and voxel channel of the fusion probability weight;

[0097] The product of the point cloud channel and the point cloud feature, and the product of the voxel channel and the voxel feature are summed to obtain the target fusion feature of the original event point cloud.

[0098] The point cloud channel is the part of the fusion probability weight related to the point cloud features, and the voxel channel is the part of the fusion probability weight related to the voxel features.

[0099] In step S104, the obtained target fusion features can be processed using a pre-built detection model based on neural network technology to obtain the drone detection result of the target scene.

[0100] The drone detection result obtained in this embodiment can be used to indicate whether there is a drone in the target scene, and further indicate the position of the drone in the target scene when a drone exists, and can further indicate the movement trajectory of the drone in the target scene within a period of time corresponding to the original event point cloud.

[0101] The construction method and structure of the above-mentioned detection model based on neural network technology can be found in relevant existing technical documents and will not be described in detail.

[0102] Optionally, the dual-branch segmentation network for extracting point cloud features and voxel features in this embodiment may have the following structure: Figure 3 The structure shown, Figure 3 The upper part is the voxel feature branch, and the lower part is the point cloud feature branch.

[0103] It can be seen that the point cloud feature branch includes four multilayer perceptrons (MLPs), which are respectively recorded as the first multilayer perceptron, the second multilayer perceptron, the third multilayer perceptron and the fourth multilayer perceptron (i.e. Figure 3 MLP1 to MLP4 are shown in the figure). Each MLP consists of a linear fit and a nonlinear activation function. The process of a single-layer MLP processing input features can be described as follows:

[0104] Xout=f(W·X ie +b) (5)

[0105] Where Xout is the output of the MLP, W and b are the weights and biases between neurons in the MLP, and function f is the activation function, which can usually be a sigmoid function or a tanh function.

[0106] The dimensions of the four MLPs can be set as needed. For example, in this embodiment, the dimensions of the first multilayer perceptron, the second multilayer perceptron, the third multilayer perceptron, and the fourth multilayer perceptron can be 32, 256, 128, and 32, respectively. That is, an event point can obtain a 32-dimensional feature vector after being processed by MLP1, a 256-dimensional feature vector after being processed by MLP2, a 128-dimensional feature vector after being processed by MLP3, and a 32-dimensional feature vector after being processed by MLP4.

[0107] The voxel feature branch includes an input layer, a first downsampling layer, a second downsampling layer, a third downsampling layer, a fourth downsampling layer, a first upsampling layer, a second upsampling layer, a third upsampling layer, and a fourth upsampling layer.

[0108] It can be seen that the structure adopted by the voxel feature branch is similar to the existing Unet network structure. It first uses an input convolution to extract contextual information from the original input, then performs four downsampling stages, and finally performs four upsampling stages to restore the original input state. The point cloud feature branch contains four layers of MLP to extract point-by-point features of different dimensions.

[0109] Specifically, Figure 3 Each black arrow in the figure represents a downsampling layer, which are the first to fourth downsampling layers from left to right. Except for the white arrow corresponding to the input layer, each white arrow represents an upsampling layer, which are the first to fourth upsampling layers from left to right. Each downsampling layer can perform convolution and downsampling on the input features, and each upsampling layer can perform convolution and upsampling on the input features.

[0110] The dimension of the input layer of the voxel feature branch can be 32, the dimensions of the first to fourth downsampling layers of the voxel feature branch are 64, 128, 256, 256 respectively, and the dimensions of the first to fourth upsampling layers of the voxel feature branch are 128, 128, 64 and 32 respectively.

[0111] In this embodiment, in order to further improve the accuracy of the detection result, Figure 3 As shown in the figure, the features output by both sides can be fused multiple times during the processing of the voxel feature branch and the point cloud feature branch, and the features after each fusion are input into the corresponding branch for further processing until the final target fusion feature is obtained. In other words, the fusion of voxel features and point cloud features can occur after the input convolution, the fourth downsampling, the second downsampling, and the last upsampling stage.

[0112] Specifically, the point cloud feature branch of the dual-branch segmentation network is used to process the original event point cloud to obtain point cloud features, which may include:

[0113] Processing the original event point cloud using a first multi-layer perceptron to obtain a first point cloud feature;

[0114] Processing the first fusion feature using a second multi-layer perceptron to obtain a second point cloud feature;

[0115] Processing the second fusion feature using a third multi-layer perceptron to obtain a third point cloud feature;

[0116] Processing the third fusion feature using a fourth multi-layer perceptron to obtain a fourth point cloud feature;

[0117] The voxel feature branch of the dual-branch segmentation network is used to process point cloud voxels to obtain voxel features, which may include:

[0118] Process the point cloud voxels using the input layer to obtain the first voxel feature;

[0119] The first fusion feature is processed in sequence by using a first downsampling layer, a second downsampling layer, a third downsampling layer, and a fourth downsampling layer to obtain a second voxel feature;

[0120] The second fusion feature is processed in sequence by the first upsampling layer and the second upsampling layer to obtain a third voxel feature;

[0121] The third fusion feature is processed in sequence by the third upsampling layer and the fourth upsampling layer to obtain a fourth voxel feature;

[0122] The first fusion feature is obtained by fusing the first point cloud feature and the first voxel feature;

[0123] The second fusion feature is obtained by fusing the second point cloud feature and the second voxel feature;

[0124] The third fusion feature is obtained by fusing the third point cloud feature and the third voxel feature;

[0125] The fourth point cloud feature and the fourth voxel feature are used to fuse to obtain the target fusion feature.

[0126] That is, the point cloud features and voxel features used for fusion in S103 may be the fourth point cloud features output by MLP4 and the fourth voxel features output by the fourth upsampling layer.

[0127] Combination Figure 3 , the fourth point cloud feature and the fourth voxel feature are obtained as follows:

[0128] Input the original event point cloud into MLP1 to obtain the first point cloud feature B1 output by MLP1, input the point cloud voxel corresponding to the original event point cloud into the input layer of the voxel feature branch to obtain the first voxel feature A1 output by the input layer;

[0129] A1 and B1 are input into the fusion module, and the fusion module is used to fuse A1 and B1 to obtain the first fusion feature;

[0130] The part S11 related to the point cloud feature in the first fusion feature is input to MLP2 to obtain the second point cloud feature B2 output by MLP2, and the part S12 related to the voxel feature in the first fusion feature is input to the first downsampling layer, and S12 is processed in sequence by the first downsampling layer to the fourth downsampling layer to obtain the second voxel feature A2 output by the fourth downsampling layer, wherein, except for the fourth downsampling layer, the output of each downsampling layer is used as the input of the next downsampling layer;

[0131] A2 and B2 are input into the fusion module, and the fusion module is used to fuse A2 and B2 to obtain the second fusion feature;

[0132] The part S21 of the second fused feature related to the point cloud feature is input to MLP3 to obtain the third point cloud feature B3 output by MLP3, and the part S22 of the second fused feature related to the voxel feature is input to the first upsampling layer, and S22 is processed in sequence by the first upsampling layer and the second upsampling layer to obtain the third voxel feature A3 output by the second upsampling layer;

[0133] A3 and B3 are input into the fusion module, and the fusion module is used to fuse A3 and B3 to obtain the third fusion feature;

[0134] The part S31 of the third fusion feature related to the point cloud feature is input into MLP4 to obtain the fourth point cloud feature B4 output by MLP4, and the part S32 of the third fusion feature related to the voxel feature is input into the third upsampling layer, and S32 is processed in sequence by the third upsampling layer and the fourth upsampling layer to obtain the fourth voxel feature A4 output by the fourth upsampling layer.

[0135] In the above processing, the fusion module fusion process of A1 and B1, the fusion process of A2 and B2, and the fusion process of A3 and B3 can be found in Figure 2 The corresponding contents in the aforementioned embodiments are not repeated here.

[0136] The training process of the above-mentioned dual-branch segmentation network can be referred to in the relevant prior art and will not be described in detail.

[0137] When training the dual-branch segmentation network, the model parameters can be randomly initialized and optimized using the Adam optimizer. Each time, 0.5s of the original event point cloud is input, the training batch size is 16, the momentum is 0.9, the decay coefficient is set to 0.0002, and the maximum number of iterations is 80000. The initial learning rate is set to 0.0002. When the number of iterations reaches 60000, the learning rate can be linearly decayed to 0.000002 to promote further convergence of the loss function. The training process can be performed on a single NVIDIA3090 GPU, and it takes about 10 hours to complete a full training.

[0138] In this embodiment, the accuracy of this embodiment can be evaluated by the evaluation indicators of the dual-branch segmentation network. Specifically, the evaluation indicators of the dual-branch segmentation network may include intersection-over-union (IoU), average precision (AP), trajectory detection rate, and false alarm rate.

[0139] The reasons for introducing the above evaluation indicators are:

[0140] This embodiment regards the event target detection task as an event point cloud segmentation task, so the point cloud IoU is used as an evaluation indicator. In order to facilitate comparison with the frame-based target detection method, this embodiment also uses AP as an evaluation indicator. Since this embodiment can directly detect the target motion trajectory, the trajectory detection rate and false alarm rate are also used as evaluation indicators.

[0141] Among them, IoU can represent the degree of overlap between the predicted segmentation result and the true label. The calculation method of IoU is to perform intersection and union operations on the predicted result and the true label area, and then calculate their ratio. The larger the IoU value, the higher the overlap between the predicted point cloud and the true value point cloud, and the better the segmentation effect. Specifically, the calculation formula of IoU can be calculated using the following formula (6).

[0142]

[0143] Among them, TP represents the correctly predicted target, FP represents the incorrectly predicted target, and FN represents the incorrectly predicted background.

[0144] AP is the area calculated based on the Precision-Recall curve. It can be calculated by interpolating the Precision at different recall rates and calculating the area under the interpolation curve. AP is a comprehensive performance indicator for evaluating target detection algorithms at different recall rates.

[0145] When the intersection-and-union ratio of the segmented event points and the true value trajectory is above 80%, the target trajectory is considered to be detected successfully and marked as the correctly predicted trajectory TP. If the noise is mistakenly detected as a trajectory instance, the target trajectory is considered to be detected incorrectly and marked as the incorrectly predicted trajectory FP. If the true trajectory is predicted as background noise, the trajectory is marked as the incorrectly predicted background FN. The trajectory detection rate Pd refers to the ratio of correctly detected trajectory instances to all true value trajectory instances, which can be specifically defined by the following formula (7).

[0146]

[0147] The trajectory false alarm rate Pf refers to the ratio of the falsely detected trajectory instances to the total true trajectory instances, which can be specifically defined by the following formula (8).

[0148]

[0149] In order to verify the detection capability of the method proposed in this embodiment for low-altitude UAV targets, the method of this embodiment and some existing networks were compared on the open source Ev-UAV dataset. The existing networks used for comparison may include: YOLOV7, SSD, FastRCNN, Pointnet, Pointnet++.

[0150] The experimental results are shown in Table 1.

[0151] Table 1

[0152]

[0153] The numerical value in the brackets of each cell in Table 1 represents the difference between the method of this embodiment and the network in the corresponding index. For example, the Pd index of the method of this embodiment is 6.7 percentage points (6.7%) higher than the Pd index of YOLOV7.

[0154] As can be seen from Table 1, on the one hand, compared with the frame-based method, the point-based method generally performs better. This is mainly because the long-range UAV target is small in size and has no obvious shape contours and other features, so it is difficult for the model to effectively detect the target, while the point cloud-based method can directly segment the target motion trajectory features, so it has better performance. On the other hand, the multi-view fusion method is relatively better than the single-view method. Despite the use of multi-view fusion, the method of this embodiment still shows very competitive running latency due to the compact network design and efficient implementation. Compared with the suboptimal method Pointnet++, the Pd of the method proposed in this embodiment is 3 percentage points lower, the false alarm rate Pf is reduced by 4.1 percentage points, the AP is increased by 2.2 percentage points, the IoU is increased by 2.8 percentage points, and the running time is shortened by 10.1 milliseconds.

[0155] In order to further verify the accuracy of the method of this embodiment, three groups of representative scenes are selected, and different detection algorithms are used to detect the same scene. The visualization results obtained by the detection are as follows: Figure 4 , Figure 5 and Figure 6 As shown, Figure 4 (1), (2) and (3) are RGB images obtained by shooting the corresponding scenes. Figure 4 (4), (5) and (6) are detected using YOLOV7 Figure 4 The detection results of drone targets obtained from the scenes corresponding to (1), (2) and (3) are as follows: Figure 5 (1), (2) and (3) are detected using PointNet++ Figure 4 The detection results of drone targets obtained from the scenes corresponding to (1), (2) and (3) are as follows: Figure 5 (4), (5) and (6) are detected by the method of this embodiment. Figure 4 The detection results of drone targets obtained from the scenes corresponding to (1), (2) and (3) are as follows: Figure 6 (1), (2) and (3) are Figure 4 The real drone targets are identified by manual annotation in the scenes corresponding to (1), (2) and (3). In the above figures, the red box is the correct detection result, the yellow box is the false alarm, and the green box is the missed detection.

[0156] It can be seen from the above schematic diagram that the frame-based algorithm YOLOV7 has missed detections and has many false alarms. The point cloud-based algorithm PointNet++ has no missed detections, but there are a small number of false alarms. The method proposed in this embodiment can effectively detect all small target drones without a false alarm rate.

[0157] In order to verify the effectiveness of multi-view fusion, this application further conducted ablation experiments on single view and dual view fusion, and the results are shown in Table 2. As can be seen from the table, compared with using only a single view, the Pd of the method proposed in this embodiment can be increased by up to 6.9 percentage points, the false alarm rate Pf can be reduced by 6.6 percentage points, and the IoU can be increased by up to 7.1 percentage points, while the reasoning time still meets the real-time detection requirements. The above shows that the view fusion method can effectively improve performance at an acceptable complexity overhead.

[0158] In Table 2, V and V+ represent different voxel resolutions, V represents 4 pixels × 4 pixels × 20 milliseconds, and V+ represents 2 pixels × 2 pixels × 10 milliseconds.

[0159] Table 2

[0160]

[0161]

[0162] In order to verify the effectiveness of the gating mechanism for multi-view fusion, this application uses different fusion methods to conduct ablation experiments, and the results are shown in Table 3. It can be seen that compared with addition, multiplication and splicing, the detection rate Pd of the gated fusion method is increased by 6.3, 4.7, and 2.7 percentage points, the false alarm rate Pf is reduced by 5.4, 4.4, and 3.1 percentage points, and the IOU is increased by 3.7, 6.1, and 5 percentage points, respectively, which shows the effectiveness of the gated fusion mechanism for multi-view fusion.

[0163] Table 3

[0164] method Pd(%) Pf(%) AP IoU(%) Add 90.2(+6.3) 8.63(-5.42) 67.3(+2.2) 66.8(+3.7) Multi 91.8(+4.7) 7.68(-4.47) 66.9(+2.6) 64.2(+6.1) Concat 93.8(+2.7) 6.38(3.17) 67.6(+1.9) 65.3(+5.0) This embodiment 96.5 3.21 69.5 70.3

[0165] In any embodiment of the present application, the method for training the above-mentioned dual-branch segmentation network may include:

[0166] 1. Process the sample event point cloud according to the dual-branch segmentation network to obtain the drone detection result of the sample event point cloud;

[0167] 2. Determine the network loss of the dual-branch segmentation network based on the drone detection results of the sample event point cloud and the preset drone annotation results of the sample event point cloud;

[0168] 3. Update the network parameters of the dual-branch segmentation network according to the network loss until the network loss meets the preset convergence conditions.

[0169] In step 1, the sample event point cloud may be an event point cloud obtained by an event camera, and the event point cloud has been previously manually annotated to determine the corresponding drone annotation result. As an example, the drone annotation result may be Figure 6 (1) to (3) represent.

[0170] In step 1, based on the dual-branch segmentation network that needs to be trained, the sample event point cloud can be detected according to the detection method provided in the above embodiment to obtain the drone detection result of the sample event point cloud.

[0171] In step 2, the drone detection result of the sample event point cloud and the drone annotation result of the preset sample event point cloud can be compared, and the network loss can be determined based on the comparison result. Specifically, the greater the difference between the drone detection result and the drone annotation result, the greater the network loss, and the smaller the difference between the drone detection result and the drone annotation result, the closer the drone detection result is to the drone annotation result, and the smaller the network loss. The network loss can be determined based on a variety of existing methods, which will not be described in detail.

[0172] In step 3, it can be determined whether the network loss obtained this time meets the convergence conditions. If it does not meet the convergence conditions, the network parameters of the two-branch segmentation network can be updated based on the network loss obtained this time. After the update is completed, steps 1 and 2 can be executed again based on the updated two-branch segmentation network. This can be repeated multiple times until the network loss obtained at a certain time meets the convergence conditions.

[0173] After executing step 2 once, if it is found that the network loss obtained this time meets the convergence condition, it can be determined that the training process of the two-branch segmentation network is completed and step 3 is no longer executed.

[0174] The specific method for updating network parameters according to network loss can be found in the relevant prior art and will not be described in detail.

[0175] The above convergence conditions can be set as needed without limitation.

[0176] As some examples, convergence conditions may include:

[0177] The network loss obtained this time is less than a preset first threshold.

[0178] The first threshold can be set according to actual conditions. For example, the first threshold can be set to 0.2. After each execution of step 2, if the obtained network loss is greater than or equal to 0.2, it is determined that the convergence condition is not met, and if it is less than 0.2, it is determined that the convergence condition is met.

[0179] In other examples, the convergence conditions may also include:

[0180] The difference between the network loss obtained this time and the network loss obtained last time is less than a preset second threshold.

[0181] The second threshold value can be set according to actual conditions. Exemplarily, the second threshold value can be set to 0.05. Based on the convergence condition, after each execution of step 2, the network loss obtained this time can be subtracted from the network loss obtained when step 2 was previously executed, and then the absolute value of the difference is taken. If the absolute value of the difference is greater than or equal to 0.05, it is determined that the network loss obtained this time does not meet the convergence condition. If the absolute value of the difference is less than 0.05, it is determined that the network loss obtained this time meets the convergence condition.

[0182] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0183] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0184] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0185] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0186] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A dual-view fusion drone detection method based on event camera detection, characterized in that: include: Obtain the original event point cloud of the target scene through the event camera; Extracting point cloud features and voxel features of the original event point cloud; Fusing the point cloud features and the voxel features to obtain target fusion features of the original event point cloud; A drone detection result of the target scene is obtained according to the target fusion feature detection, and the drone detection result is used to indicate a drone in the target scene.

2. The method according to claim 1, characterized in that The fusing the point cloud features and the voxel features to obtain the target fusion features of the original event point cloud includes: Performing convolution processing and normalization processing on a spliced ​​feature formed by splicing the point cloud feature and the voxel feature to obtain a fusion probability weight; The point cloud features and the voxel features are weightedly summed according to the fusion probability weights to obtain target fusion features of the original event point cloud.

3. The method according to claim 2, characterized in that The step of performing weighted summation of the point cloud features and the voxel features according to the fusion probability weight to obtain the target fusion features of the original event point cloud includes: Separating the point cloud channel and the voxel channel of the fusion probability weight; The product of the point cloud channel and the point cloud feature and the product of the voxel channel and the voxel feature are summed to obtain the target fusion feature of the original event point cloud.

4. The method according to claim 1, characterized in that: The step of extracting point cloud features and voxel features of the original event point cloud comprises: Processing the original event point cloud using the point cloud feature branch of the dual-branch segmentation network to obtain point cloud features; The original event point cloud is discretized to obtain point cloud voxels, and the point cloud voxels are processed using a voxel feature branch of a dual-branch segmentation network to obtain voxel features.

5. The method according to claim 4, characterized in that The point cloud feature branch includes a first multi-layer perceptron, a second multi-layer perceptron, a third multi-layer perceptron and a fourth multi-layer perceptron; The voxel feature branch includes an input layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a fourth down-sampling layer, a first up-sampling layer, a second up-sampling layer, a third up-sampling layer and a fourth up-sampling layer.

6. The method according to claim 5, characterized in that The point cloud feature branch of the dual-branch segmentation network is used to process the original event point cloud to obtain point cloud features, including: Processing the original event point cloud using the first multi-layer perceptron to obtain a first point cloud feature; Processing the first fusion feature using the second multi-layer perceptron to obtain a second point cloud feature; Processing the second fusion feature using the third multi-layer perceptron to obtain a third point cloud feature; Processing the third fusion feature using the fourth multi-layer perceptron to obtain a fourth point cloud feature; The step of processing the point cloud voxels using the voxel feature branch of the dual-branch segmentation network to obtain voxel features includes: Processing the point cloud voxels using the input layer to obtain first voxel features; Processing the first fusion feature in sequence by using the first downsampling layer, the second downsampling layer, the third downsampling layer and the fourth downsampling layer to obtain a second voxel feature; Using the first upsampling layer and the second upsampling layer to sequentially process the second fusion feature to obtain a third voxel feature; Using the third upsampling layer and the fourth upsampling layer to sequentially process the third fusion feature to obtain a fourth voxel feature; The first fusion feature is obtained by fusing the first point cloud feature and the first voxel feature; The second fusion feature is obtained by fusing the second point cloud feature and the second voxel feature; The third fusion feature is obtained by fusing the third point cloud feature and the third voxel feature; The fourth point cloud feature and the fourth voxel feature are used to fuse to obtain the target fusion feature.

7. The method according to claim 4, characterized in that The evaluation indicators of the dual-branch segmentation network include overlap, average accuracy, trajectory detection rate and false alarm rate.

8. The method according to claim 4, characterized in that The method of training the dual-branch segmentation network includes: Processing the sample event point cloud according to the dual-branch segmentation network to obtain a drone detection result of the sample event point cloud; Determining the network loss of the dual-branch segmentation network according to the drone detection result of the sample event point cloud and the preset drone labeling result of the sample event point cloud; The network parameters of the dual-branch segmentation network are updated according to the network loss until the network loss meets a preset convergence condition.

9. The method according to claim 8, characterized in that The convergence conditions include: The network loss obtained this time is less than a preset first threshold; or, The difference between the network loss obtained this time and the network loss obtained last time is less than a preset second threshold.

10. The method according to claim 1, characterized in that The step of extracting point cloud features and voxel features of the original event point cloud comprises: Dividing the original event point cloud into a plurality of basic processing units using a time window with a fixed time interval; Extracting point cloud features and voxel features of at least one of the basic processing units.

Images