All-day multi-scale low-small slow target detection method combining visible light and infrared
By combining a multi-scale single-frame target detection network and a trajectory detection network, the problem of large scale variations and difficulty in detecting small, slow targets in images at all times is solved. This achieves all-day multi-scale detection of small, slow targets, which features simple parameter tuning, wide applicability, and strong robustness.
Patent Information
- Application Number
- CN202311250504.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-26
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-09-26
AI Technical Summary
Existing technologies cannot adapt to the large scale variations of small and slow targets in images and are difficult to detect at all times of day. Furthermore, the separate design of visible light and infrared methods leads to complex parameter tuning of the algorithms.
A multi-scale single-frame target detection network and trajectory detection network combining visible light and infrared are adopted. Combined with the spatiotemporal information of UAV movement, target detection is performed through the multi-scale single-frame target detection network model and trajectory detection network to achieve all-day, multi-scale, low, small, and slow target detection.
It achieves all-day, multi-scale detection of small, slow-moving targets, with simple parameter tuning, wide applicability, and strong robustness, demonstrating more significant advantages in complex and long-distance target scenarios.
Smart Images
Figure CN117197696B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image data processing, and particularly relates to a full-time multiscale low-small-slow target detection method combining visible light and infrared. BACKGROUND
[0002] In the scenario of monitoring low-small-slow targets such as unmanned aerial vehicles, according to the distance of the target to be detected, the target presents different scales in the monitoring image. When the target is close to the observation platform, the target has obvious morphological characteristics, and a commonly used single-frame detection algorithm can achieve good detection effect. However, when the target is far away, the target appears as a weak small point source target in the image, and it is difficult to distinguish the target from noise by using only the single-frame detection algorithm. On the other hand, in the case of dark ambient light, the visible light camera is difficult to collect target information, and the infrared camera can capture the infrared radiation signal of the low-small-slow target in the dark environment. Combining the visible light camera and the infrared camera can achieve full-time detection of the target. However, the existing methods often design target detection algorithms for visible light or infrared images respectively, and two sets of algorithms need to be designed and trained respectively, which is complex.
[0003] The existing technology has the following two problems:
[0004] (1) For low-small-slow targets, due to the large flight range, the target has scale changes in the image, and the existing method cannot adapt to such large-scale target changes, resulting in poor detection effect.
[0005] (2) The existing visible light and infrared methods often design two different algorithms to detect the target, which leads to difficult algorithm parameter adjustment and algorithm selection difficulty.
[0006] (2) The existing visible light and infrared methods often design two different algorithms to detect the target, which leads to difficult algorithm parameter adjustment and algorithm selection difficulty. SUMMARY
[0007] The technical problem to be solved by the present application is to provide a full-time multiscale low-small-slow target detection method combining visible light and infrared, which solves the problems of large scale changes of low-small-slow targets in the image plane and the complexity of detection algorithm design caused by the large difference between visible light cameras and infrared cameras in full-time scenes.
[0008] To solve the above technical problems, the technical scheme adopted by the present application is as follows: a full-time multiscale low-small-slow target detection method combining visible light and infrared, comprising the following steps:
[0009] S1, acquiring an infrared sequence image or a visible light sequence image containing a low-small-slow target;
[0010] S2, input the infrared sequence image or the visible light sequence image into a multi-scale single-frame target detection network model, extract a target detection result of each frame of image in the infrared sequence image or the visible light sequence image, that is, an alternative point, and all the alternative points in the time domain are constructed into a point cloud space;
[0011] S3, input the point cloud space into a trajectory detection network to obtain a detection result; wherein the trajectory detection network comprises a feature extraction layer, a feature aggregation layer and a result output module connected in sequence; the feature extraction layer is used for extracting local features of each alternative point, the feature aggregation layer aggregates the local features of all the alternative points into global features, and the result output module splices the local features and the global features of each alternative point to form final features of each alternative point, and the final features are input into a plurality of fully connected layers to obtain the detection result.
[0012] The present application combines a multi-scale single-frame target detection network model and a trajectory detection network, fully utilizes the space-time information of the unmanned aerial vehicle movement, can be applied to visible light images and infrared images at the same time, can train the two kinds of data mixed together, and is convenient for method adjustment. The present application realizes all-weather multi-scale low-small-slow target detection, and has the characteristics of simple and convenient parameter adjustment, wide application range and strong robustness.
[0013] In the present application, the multi-scale single-frame target detection network model comprises a plurality of groups of cascaded convolution layers, and adjacent two groups of convolution layers are connected through a maximum pooling layer; each group of convolution layers comprises at least one convolution layer.
[0014] In the present application, the output of the result output module is an N*2 matrix, each row of the matrix represents an alternative point, and there are two columns, the first column represents the confidence of the alternative point being a real target, and the second column represents the track number of the target; when the confidence is higher than a set threshold, the target is considered to be a real target; N represents the number of alternative point targets.
[0015] In the present application, in order to improve the detection accuracy, it further comprises: eliminating the target with a confidence lower than the set threshold.
[0016] In the present application, in order to facilitate viewing of the detection result, it further comprises: visualizing the target with a confidence higher than the set threshold.
[0017] In the present application, the feature extraction layer comprises a plurality of cascaded fully connected layers.
[0018] In the present application, the number of fully connected layers in the feature extraction layer is 5; the input of the feature extraction layer is an N*M matrix, N represents the number of alternative point targets, and M represents the number of feature information of the alternative point.
[0019] In the present application, the feature aggregation layer is a maximum pooling layer.
[0020] In the present application, the number of fully connected layers in the result output layer is 3.
[0021] Compared with the prior art, the present application has the beneficial effects that: the present application makes full use of the space-time information of unmanned aerial vehicle movement, and shows more obvious advantages in complex scenes and long-distance target scenes, and the method of the present application can be applied to visible light images and infrared images at the same time, and the two kinds of data can be mixed together for training, which is convenient for method adjustment. The present application realizes all-weather multi-scale low-small-slow target detection, and has the characteristics of simple and convenient parameter adjustment, wide application range and strong robustness. BRIEF DESCRIPTION OF DRAWINGS
[0022] Figure 1 A flow chart of the multi-scale space-time fusion detection method for unmanned aerial vehicles of the embodiment of the present application is shown in the figure.
[0023] Figure 2 A single-frame target detection network schematic diagram of the embodiment of the present application is shown in the figure.
[0024] Figure 3 A multi-frame candidate point schematic diagram (dimension reduction) of the embodiment of the present application is shown in the figure.
[0025] Figure 4 A trajectory detection network schematic diagram of the embodiment of the present application is shown in the figure.
[0026] Figure 5 A real target motion trajectory diagram of the embodiment of the present application is shown in the figure.
[0027] Figure 6 A low-small-slow target detection model training loss trend of the embodiment of the present application is shown in the figure.
[0028] Figure 7 Unmanned aerial vehicle detection effect under different scenes of the embodiment of the present application is shown in the figure, (a) unmanned aerial vehicle relative size 782*411, (b) unmanned aerial vehicle relative size 5*5, (c) dark environment. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0030] In this document, the terms "first", "second", and other similar terms are not intended to imply any order, quantity, and importance, but are merely used to distinguish different elements. In this document, the terms "one", "a", and other similar terms are not intended to mean that there is only one of the described things, but that the description is directed to only one of the described things, which can have one or more. In this document, the terms "include", "comprise", and other similar terms are intended to mean logical relationships, and cannot be regarded as indicating spatial structural relationships. For example, "A includes B" is intended to mean that B logically belongs to A, and does not mean that B is located inside A in space. In addition, the meaning of the terms "include", "comprise", and other similar terms should be regarded as open, rather than closed. For example, "A includes B" is intended to mean that B belongs to A, but B does not necessarily constitute all of A, and A can also include C, D, E, and other elements.
[0031] Embodiment 1
[0032] Embodiment 1 of the present application provides a joint visible light and infrared all-day multi-scale low small slow target detection method, which includes two stages, suspected target point detection (output target candidate points) and trajectory detection stage (detect target trajectory in space-time), as shown in Figure 1
[0033] The specific implementation steps of Embodiment 1 of the present application include:
[0034] Step 1, obtain an infrared sequence image or a visible light sequence image containing a low small slow target;
[0035] Step 2, input the infrared sequence image or the visible light sequence image into a multi-scale single-frame target detection network model;
[0036] Step 3, trajectory level detection stage, according to the result of step 2, use the topological properties of the trajectory to judge the authenticity of the target, eliminate false targets, and finally output the detection result.
[0037] In the embodiment of the present application, the low small slow target can be a UAV target.
[0038] As shown in Figure 2 The input of the single-frame detection network of the embodiment of the present application is an image with a length of H, a width of W, and a channel number of 3. After feature extraction by the Backbone part, detection will be performed on four different depth feature maps. In the shallower layer, the model can capture the edge and other shallow information of the target. With the deepening of the model depth, the receptive field gradually becomes larger, and the model can capture deeper features. Figure 3 Conv is a convolutional layer, and MaxPool is a max-pooling layer. The final network will detect the target on four scale feature maps, thereby enhancing the detection performance of multi-scale targets.
[0039] In this stage, the output of the network is the detection frame of the target, the center point of the detection frame represents the position of the target, denoted as Q=(x, y), x and y respectively correspond to the coordinates of the center position of the target on the image plane, the area of the detection frame is denoted as S, the length-width ratio of the detection frame is denoted as R, the target category is denoted as C, and the confidence is denoted as P. The final detection result of the target is represented as (Q, S, R, C, P), which can be regarded as a point in a five-dimensional space, and this point is called an alternative point. There can be multiple alternative points in each frame of image, and all the alternative points in the time domain are modeled as a point cloud space, as shown in Figure 3 .
[0040] As shown in Figure 4 , in the embodiment of the present application, the input of the trajectory detection network is the point cloud in step 2, and the trajectory is represented as an N*6 matrix, N represents the number of alternative point targets in the trajectory, and 6 corresponds to the feature information (Q, S, R, C, P) of the alternative point, wherein Q=(x, y). The network input successively passes through a feature extraction layer, a feature aggregation layer, and a result output module, finally judges whether the trajectory is a real target, and outputs the detection result. The input of the feature extraction layer is an N*6 matrix, which first passes through two fully connected layers, and the feature is raised from 6 dimensions to 32 dimensions and then to 64 dimensions. This operation raises the feature of each point from 6 dimensions to 64 dimensions, and this 64-dimensional feature contains the local feature of each point. Then, it passes through three fully connected layers again, and the feature is raised from 64 dimensions to 256 dimensions to 512 dimensions to 1024 dimensions; the feature aggregation layer is a maximum pooling operation, which aggregates N*1024 data into a 1*1024 global feature; the result output module splices the 64-dimensional local feature of each point and the 1*1024-dimensional global feature to form the final feature of each point, and then passes through a fully connected layer again, from N*1088 (64+1024) to N*512 to N*128, and finally to N*2, to obtain the detection result.
[0041] The trajectory detection principle is explained as follows: a moving target will exhibit continuous motion in the time domain, and the center point Q of the target, the area S of the detection frame, the length-width ratio R of the detection frame, the target category C, and the confidence P will not change dramatically between adjacent frames, but will change in a certain rule, showing a continuous curve, while noise is randomly generated and cannot be associated as a regular motion trajectory. Even if it is associated as a trajectory, the trajectory cannot exhibit regular motion characteristics.
[0042] The output of the trajectory detection network is an N*2 matrix, each row of the matrix represents an alternative point, and there are two columns, the first column represents the confidence that the alternative point is a real target, and the second column represents the trajectory number of the target, for example, a point is (0.6, 1), which means that the probability that the point is a real target is 0.6, and it belongs to the first trajectory. A confidence threshold is set, when the confidence is higher than the threshold, it is considered that the target is a real target, and all alternative points with high confidence are visualized, such as Figure 5 In the embodiment of the application, the confidence threshold is set to 0.5.
[0043] In the embodiment of the application, 17343 real unmanned aerial vehicle data are collected, of which 8654 are outdoor unmanned aerial vehicle data and 8689 are indoor unmanned aerial vehicle data. The data set is divided according to the ratio of 8:1:1, and the specific data set division is shown in Table 1.
[0044] Table 1 Details of data set division
[0045]
[0046] On the above data set, the designed model is trained, and the parameter settings are as shown in Table 2:
[0047] Table 2 Training parameters of unmanned aerial vehicle recognition and positioning model
[0048]
[0049] The training results are as shown in Figure 6 and Table 3:
[0050] Table 3 Model evaluation index effect
[0051]
[0052] Part of the data visualization results are as shown in Figure 7 . Figure 7 In the embodiment of the application, (a) is the detection result of the unmanned aerial vehicle image at a close distance and a large scale; (b) is the detection result of the small scale under a long distance; and (c) is the target detection result under infrared camera imaging. It can be seen that the model has good detection effect on small and slow targets at large and small scales, and also has good detection effect on visible light images and infrared images, proving that the model has detection capability for all-weather multi-scale small and slow targets.
[0053] In the embodiment of the application, the spatial target detection is combined, the time domain continuity of the moving target is used to detect the weak and small point target, and the simultaneous detection of targets of different sizes at different distances is realized.
[0054] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.
[0055] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.
Claims
1. A method for all-day multi-scale low-small-slow target detection combining visible light and infrared, characterized in that, The method comprises the following steps: S1, acquiring an infrared sequence image or a visible light sequence image containing small and slow targets; S2, inputting the infrared sequence image or the visible light sequence image into a multi-scale single-frame target detection network model to extract a target detection result of each frame of image in the infrared sequence image or the visible light sequence image, i.e. an alternative point, and constructing all alternative points in the time domain into a point cloud space; the output of the multi-scale single-frame target detection network model is a detection frame of a target, the center point of the detection frame represents the position of the target, denoted as Q=(x, y), x and y respectively correspond to the coordinates of the center position of the target on the image plane, the area of the detection frame is denoted as S, the length-width ratio of the detection frame is denoted as R, the target category is denoted as C, and the confidence is denoted as P, and finally the detection result of the target is represented as (Q, S, R, C, P), which can be regarded as a point in a five-dimensional space, and the point is referred to as an alternative point; S3, inputting the point cloud space into a track detection network to obtain a detection result; wherein the track detection network comprises a feature extraction layer, a feature aggregation layer and a result output module connected in sequence; the feature extraction layer is used for extracting local features of each alternative point, the feature aggregation layer aggregates the local features of all alternative points into global features, and the result output module splices the local features and the global features of each alternative point to form final features of each alternative point, and the final features pass through a plurality of fully connected layers to obtain the detection result; the input of the track detection network is the point cloud in S2, the track is represented as an N*6 matrix, N represents the number of alternative point targets in the track, and 6 corresponds to the feature information (Q, S, R, C, P) of the alternative point, wherein Q=(x, y); the track detection network input sequentially passes through the feature extraction layer, the feature aggregation layer and the result output module, finally judges whether the track is a real target, and outputs the detection result; the input of the feature extraction layer is an N*6 matrix, which first passes through two fully connected layers, and the features are upgraded from 6 dimensions to 32 dimensions and then to 64 dimensions, so that the features of each point are upgraded from 6 dimensions to 64 dimensions, and the 64-dimensional features contain the local features of each point, and then pass through three fully connected layers, and the features are upgraded from 64 dimensions to 256 dimensions, then to 512 dimensions and finally to 1024 dimensions; the feature aggregation layer is a maximum pooling operation, which aggregates N*1024 data into a 1*1024 global feature; the result output module splices the 64-dimensional local features of each point and the 1*1024-dimensional global features to form the final features of each point, and then passes through fully connected layers again, from N*1088 (64+1024) to N*512 to N*128, and finally to N*2, to obtain the detection result. 2.The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 1, characterized in that, The multi-scale single-frame target detection network model comprises a plurality of groups of cascaded convolution layers, and adjacent two groups of convolution layers are connected through a maximum pooling layer; each group of convolution layers comprises at least one convolution layer. 3.The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 1, characterized in that, The output of the result output module is a matrix of N*2, each row of the matrix represents an alternative point, and there are two columns, the first column represents the confidence that the alternative point is a real target, and the second column represents the track number of the target; when the confidence is higher than a set threshold, the target is considered to be a real target; N represents the number of alternative point targets.
4. The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 3, characterized in that, Further comprising: Eliminate targets with a confidence lower than the set threshold.
5. The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 3, characterized in that, Further comprising: Visualize targets with a confidence higher than the set threshold. 6.The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 1, characterized in that, The feature extraction layer includes multiple cascaded fully connected layers.
7. The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 6, characterized in that, The number of fully connected layers in the feature extraction layer is 5; the input of the feature extraction layer is a matrix of N*M, N represents the number of alternative point targets, and M represents the number of feature information of the alternative point. 8.The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 1, characterized in that, The feature aggregation layer is a max-pooling layer. 9.The combined visible and infrared all-weather multi-scale low-small-slow target detection method according to claim 1, characterized in that, The number of fully connected layers in the result output layer is 3.