A method and device for detecting a moving small target in a high-altitude bird's-eye view scene

Abstract

The application discloses a high-altitude overflight scene motion small target detection method and device, including the following steps: A: constructing a to-be-recognized image sequence set; B: constructing a visual perception module, and constructing a smooth feature map sequence; C: constructing a change perception module, and obtaining a first space-time dynamic feature map corresponding to an intermediate frame in the smooth feature map sequence; D: constructing a motion information calculation module, and obtaining an optimized second space-time dynamic feature map after motion energy calculation, receptive field enlargement, motion target scale screening and background suppression; E: constructing a receptive field enlargement module, and obtaining a third space-time dynamic feature map; F: respectively obtaining corresponding third space-time dynamic feature maps, and taking the position of the maximum pixel point value in the third space-time dynamic feature map as the position of the motion small target. The application can effectively distinguish the background environment and the small target in the image, detect the position of the motion small target in the background environment, and improve the motion small target detection efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and apparatus for detecting small moving targets in aerial view scenes based on the bird's visual pathway. Background Technology

[0002] In the field of computer vision, moving small object detection has wide applications in defense, surveillance, and road safety. However, because small objects (usually defined as targets whose dimensions are 0.1 times the original image size, or targets smaller than 32×32 pixels) occupy too few pixels, physical cues are limited, and the background environment is extremely cluttered, effectively detecting moving small objects in chaotic moving backgrounds has become a huge challenge in the field of computer vision.

[0003] In nature, the biological brain possesses a remarkable ability to perceive the external environment, enabling efficient autonomous decision-making and long-term stable operation even in complex conditions. The visual system of living organisms receives a massive amount of information daily, and when faced with complex scenes, it employs a unique information processing method: rapidly focusing attention on a few visually salient objects and assigning them higher processing priority, thereby eliminating interference from other areas and reducing the complexity of information processing. Flying animals exhibit extremely high sensitivity to movement, especially movement relative to their environment; birds are particularly adept at this in high-altitude, bird-like scenarios.

[0004] Research on computational models of motor neurons in biological visual systems has been ongoing for several years. Studies have shown that a special type of neuron in the insect brain makes insects highly sensitive to small moving targets; this type of neuron is called a Small Target Motion Detector (STMD). With a more complete understanding of the electrophysiology of STMD neurons, researchers have proposed quantitative models based on STMD, such as the ESTMD model and the DSTMD model. These models can detect small moving targets against cluttered backgrounds, but they cannot distinguish between small targets and background features.

[0005] In the insect visual system, multiple visual cues are extracted by different neural circuits, and these circuits can work together to detect moving small targets. For example, large monopolar cells (LMCs) extract motion information from brightness signals, while amacrine cells (AMCs) and their downstream neurons form a contrast pathway to extract directional contrast from brightness signals. The STMD+ model utilizes the characteristics of LMC and AMC pathways to combine directional contrast and motion information, enabling more robust differentiation between moving small targets and background features. However, this method requires background motion speed and direction as prior knowledge input to the model. Without prior knowledge, this method cannot calculate the relative motion between complex backgrounds and small targets, thus failing to detect moving small targets.

[0006] Traditional motion detection methods, such as optical flow, background subtraction, and temporal difference, have evolved to detect normal-sized objects, such as pedestrians and vehicles. They utilize physical features, including shape, color, and texture, to segment regions corresponding to moving objects in the background. However, for objects as small as one or a few pixels, traditional motion detection methods are ineffective because they struggle to identify the physical features of such small objects. Furthermore, when applying background motion compensation, small targets may be obscured by pixel errors, making traditional moving object detection methods unsuitable for cluttered moving backgrounds.

[0007] In existing technologies, research and applications of moving small target detection mainly focus on infrared images. These infrared-based methods heavily rely on significant temperature differences between the background and the object of interest (such as rockets, jets, and missiles). However, such significant temperature differences are rare in nature. Furthermore, the detection environments for these methods are primarily the sky or ocean, where the scenes are relatively clear and uniform. However, these methods are not suitable for complex environments with large formats, multiple textures, and multiple colors in high-altitude overhead views, and they cannot meet the requirements of small size and low power consumption in practical applications. Summary of the Invention

[0008] The purpose of this invention is to provide a method and apparatus for detecting small moving targets in aerial view scenes based on the bird's visual pathway. This method can effectively distinguish between the background environment and small targets in an image, detect the position of small moving targets in the background environment, and improve the efficiency of small moving target detection.

[0009] The present invention adopts the following technical solution:

[0010] A method for detecting small moving targets in a high-altitude, top-down scene includes the following steps:

[0011] A: Acquire continuous images of a high-altitude, top-down scene, and sequentially use each frame of the continuous images as an intermediate frame. Combine the intermediate frame and the T frames before and after it to form an image sequence a to be recognized. i Finally, a set of image sequences to be identified is obtained, A = {a1, a2, ..., a...} N}, i = 1, 2, ..., N, where N is the total number of frames in the continuous images;

[0012] B: Construct a visual perception module and use the visual perception module to identify the image sequence a. i Gaussian smoothing is applied to the pixel values ​​of each pixel in each frame of the image to obtain a smoothed feature map with precise location information for each frame. This smoothed feature map is then used to identify the image sequence a. i The smoothed feature maps corresponding to each frame of the image constitute a smoothed feature map sequence A. p ;

[0013] In this invention, the visual perception module uses a two-dimensional Gaussian filter to simulate the function of retinal neurons, smoothing the pixel value of each pixel in each frame of the input image, in order to cooperate with subsequent modules to realize the detection of small moving targets.

[0014] C: Construct a change-aware module to process the smooth feature map sequence A. p Spatial and temporal features are extracted from the smoothed feature map to obtain the smoothed feature map sequence A. p intermediate frame N p The corresponding first spatiotemporal dynamic feature map;

[0015] D: Construct a motion information calculation module, and use the motion information calculation module to perform motion energy calculation, receptive field enlargement, motion target scale screening and background suppression on the first spatiotemporal dynamic feature map obtained in step C, to obtain an optimized second spatiotemporal dynamic feature map;

[0016] E: Construct a receptive field enlargement module, and use the receptive field enlargement module to enlarge the receptive field of the second spatiotemporal dynamic feature map obtained in step D, and finally obtain the third spatiotemporal dynamic feature map;

[0017] F: Following the methods in steps B to E, sequentially identify each image sequence a in the image sequence set A. i The process is performed to obtain the corresponding third spatiotemporal dynamic feature maps, and the position where the pixel value in the third spatiotemporal dynamic feature map reaches the maximum value is taken as the position of the moving small target.

[0018] In step B, let a be the image sequence to be identified for moving small target detection. i The duration is 2T+1 frames, for the image sequence to be identified a iInput any image in the dataset. After passing through the visual perception module, a smooth feature map P(x,y,t) with precise location information is output.

[0019]

[0020] Where x and y represent the horizontal and vertical coordinates of a pixel, respectively, and t represents time. Let represent the set of real numbers, u represent the integration variable corresponding to x, and v represent the integration variable corresponding to y. This represents the first Gaussian kernel function. σ1 represents the standard deviation of the first Gaussian kernel function;

[0021] Finally, using the image sequence to be identified, a i The 2T+1 smoothed feature maps obtained after Gaussian smoothing of all images form a smoothed feature map sequence A. p .

[0022] In step C, the change-sensing module consists of m multi-channel spatiotemporal three-dimensional filter banks with different preference directions; through the change-sensing module, the smooth feature map sequence A is processed. p intermediate frame N p Spatial feature extraction is performed, and the smoothed feature map sequence A is... p Except for intermediate frame N p The temporal information of the remaining frames is integrated into the intermediate frame N. p Temporal feature extraction is performed, ultimately yielding a smooth feature map sequence A. p Inner intermediate frame N p The corresponding first spatiotemporal dynamic feature map.

[0023] In step C, let Q represent m multi-channel spatiotemporal three-dimensional filter banks in the change sensing module, and let F be one of the multi-channel spatiotemporal three-dimensional filter banks. j The value of j ranges from {1,2,…,m}, and the intermediate frame N is calculated through convolution. p The response value of each pixel on the corresponding smooth feature map P1(x,y,t) in the preference direction θ is used to obtain the intermediate frame N. p The corresponding first spatiotemporal dynamic feature map

[0024]

[0025]

[0026] in, This represents the first spatiotemporal dynamic feature map. Represents a spatiotemporal three-dimensional filter;

[0027] Each preference direction θ corresponds to a And as a multi-channel spatiotemporal three-dimensional filter bank F j A channel, multiple preference directions θ corresponding to multiple Together they form the corresponding multi-channel spatiotemporal three-dimensional filter bank F j ; The phase offset of the spatiotemporal three-dimensional filter is given by F in a multi-channel spatiotemporal three-dimensional filter bank. j middle, For a given value, m have different values. Multi-channel spatiotemporal three-dimensional filter bank F j This constitutes Q. For matrix multiplication calculation, It is a two-dimensional Gabor filter, representing a three-dimensional spatiotemporal filter. For the two-dimensional spatial part of P1(x,y,t), perform a convolution operation on the spatial part P1(x,y).

[0028] x′(θ) = xcosθ + ysinθ;

[0029] y'(θ) = -xsinθ + ycosθ;

[0030]

[0031] Where x′(θ) and y′(θ) are the coordinates of x and y in the preference direction θ, respectively, and γ, σ², and λ represent the two-dimensional Gabor filter. The spatial aspect ratio, standard deviation, and wavelength are given, and i(t) is the temporal kernel function for detecting pixel changes.

[0032] Step D includes the following specific steps:

[0033] D1: Construct a motion energy calculation submodule and use the motion energy calculation submodule to calculate the net motion energy feature map of each pixel in the first spatiotemporal dynamic feature map in each preferred direction;

[0034] D2: Construct a receptive field enlargement submodule and use it to modify the net kinetic energy feature map. The receptive field is enlarged, and the second motion energy feature map E′ after the receptive field enlargement is finally obtained. θ (x,y,t);

[0035]

[0036] Where k1 is the size of the first pooling window. This is a max pooling operation;

[0037] E′ after the receptive field is enlarged θ (x,y,t) can detect the motion of targets at various scales. Since this application needs to extract small moving targets, further scale filtering is required to extract small moving targets.

[0038] D3: Construct a motion target scale filtering submodule, and use the motion target scale filtering submodule to perform scale filtering on the motion features in the second motion energy feature map to obtain the scale-filtered third motion energy feature map;

[0039] S θ (x,y,t)=[∫∫E′ θ (u,v,t)W s (xu,yv)dudv] + ;

[0040] Among them, S θ (x,y,t) is the third motion energy characteristic map, W s (·) denotes the lateral inhibition scale-selective kernel function, [ ] + For linear rectified functions, u represents the integral variable corresponding to x, and v represents the integral variable corresponding to y;

[0041] D4: Construct a background suppression submodule and use it to suppress the background of the third motion energy feature map;

[0042] Anomaly detection is one of the core problems in data mining. Commonly used unsupervised anomaly detection methods include statistical methods, clustering, and isolated forest methods. This invention uses statistical methods for anomaly detection. In statistics, the normality of data is assumed to follow a statistical model, and outliers are abnormal observations that are far removed from other values. This invention uses the calculation of the Z-score of observation points as an example to describe the background suppression method based on the statistical model:

[0043] The third motion energy characteristic diagram S in any direction θ (x,y,t), calculate the standard deviation σ of its distribution. f ;

[0044]

[0045] Among them, the third motion energy characteristic diagram S θ The dimensions of (x,y,t) are H×W, and mean(·) is calculated by taking the average value. The Z-score of the corresponding pixel (x,y,t) is:

[0046]

[0047] The threshold for the Z-score is set to ε, and in this embodiment, ε = 3*σ f Pixels less than or equal to the Z-score threshold ε are considered to conform to the background model distribution, while pixels greater than the Z-score threshold ε are considered outliers, i.e., target points. The final output image after background suppression is the optimized second spatiotemporal dynamic feature map O. θ (x,y,t) is:

[0048] O θ (x,y,t)=S θ (x, y, t) × [Zscore] θ (x,y,t)-ε] + .

[0049] Step D1 includes the following specific steps:

[0050] D11: The phase shift difference in m multi-channel spatiotemporal three-dimensional filter banks Q is... The two multi-channel spatiotemporal three-dimensional filter banks are grouped together and designated as F. g and F h , will F g and F h Filters with the same bias direction θ are considered as a set of orthogonal filters, and the two first spatiotemporal dynamic feature maps generated accordingly are considered as a set of orthogonal first spatiotemporal dynamic feature maps; the first motion energy feature map E is calculated from the set of orthogonal first spatiotemporal dynamic feature maps. θ (x,y,t) is the square root of the sum of squares of a set of orthogonal first-order spatiotemporal dynamic feature maps:

[0051]

[0052] D12: Based on the first motion energy feature map calculated in step D11, calculate the net motion energy feature map of each pixel in the first motion energy feature map in each preferred direction using the following formula:

[0053]

[0054] in, Represents the net kinetic energy characteristic diagram, E flk The flicker energy is represented by the average value of the output energy. M represents the number of possible values ​​for the preference direction θ.

[0055] In step E, the max pooling method is used. The process of simulating the receptive field increasing again is used to obtain the third spatiotemporal dynamic feature map O′. θ (x,y,t);

[0056]

[0057] in, For max pooling, k2 is the size of the second pooling window.

[0058] In step F, the third spatiotemporal dynamic feature map corresponding to different preference directions θ of pixels in each frame is calculated. The position of the pixel with the maximum value after summing the third spatiotemporal dynamic feature maps corresponding to all preference directions is the position (x) of the moving small target in the current frame. s ,y s );

[0059]

[0060] Where, x s y s , respectively, are the x and y coordinates of the moving small target detected at time t.

[0061] A device for detecting small moving targets in a high-altitude, top-down view scenario includes an image acquisition module, a visual perception module, a change perception module, a motion information calculation module, a receptive field enlargement module, and a target position recognition module.

[0062] The image acquisition module is used to acquire continuous images of a high-altitude, top-down scene and generate a set A of image sequences to be identified, A = {a1, a2, ..., a...}. N}, i = 1, 2, ..., N, where N is the total number of frames in the continuous images;

[0063] The visual perception module is used to acquire the image sequence a to be recognized. i Each frame of the image contains a smoothed feature map with precise location information, and the image sequence to be identified is a... i The smoothed feature maps corresponding to each frame of the image constitute a smoothed feature map sequence A. p ;

[0064] The change-aware module is used to process the smooth feature map sequence A. p Spatial and temporal features are extracted from the smoothed feature map to obtain the smoothed feature map sequence A. p intermediate frame N p The corresponding first spatiotemporal dynamic feature map;

[0065] The motion information calculation module is used to sequentially perform motion energy calculation, receptive field enlargement, moving target scale screening and background suppression on the first spatiotemporal dynamic feature map to obtain the optimized second spatiotemporal dynamic feature map.

[0066] The motion information calculation module includes a motion energy calculation submodule, a receptive field enlargement submodule, a moving target scale filtering submodule, and a background suppression submodule;

[0067] The motion energy calculation submodule is used to calculate the net motion energy feature map of each pixel in the first spatiotemporal dynamic feature map in each preferred direction;

[0068] The receptive field enlargement submodule is used to enlarge the receptive field of the net motion energy feature map to obtain a second motion energy feature map with enlarged receptive field.

[0069] The motion target scale filtering submodule is used to perform scale filtering on the motion features in the second motion energy feature map to obtain the scale-filtered third motion energy feature map.

[0070] The background suppression submodule is used to suppress the background of the third motion energy feature map; the final output map after background suppression is the optimized second spatiotemporal dynamic feature map.

[0071] The receptive field enlargement module is used to enlarge the receptive field of the second spatiotemporal dynamic feature map a second time, and finally obtain the third spatiotemporal dynamic feature map.

[0072] The location recognition module is used to sequentially acquire each image sequence a in the set of image sequences to be recognized, A. i The corresponding third spatiotemporal dynamic feature map is used, and the position where the pixel value in the third spatiotemporal dynamic feature map reaches the maximum value is taken as the position of the moving small target.

[0073] This invention constructs a method and device for detecting small moving targets in high-altitude overhead scenes by simulating the structure and perceptual characteristics of the retinal-otonic-circular process visual pathway in birds. It employs a multi-channel three-dimensional spatiotemporal Gabor filter to perceive changes in image sequences, extracting the spatiotemporal distribution characteristics of each pixel in a single calculation. This helps the model retain the motion features of small targets, effectively distinguishing between the background environment and small targets in the image, detecting the position of small moving targets in the background environment, and improving the efficiency of small moving target detection. The receptive field enlargement module designed in this invention effectively enhances the model's ability to extract invariant features of small moving targets.

[0074] This invention targets the background motion characteristics in high-altitude overhead scenes and uses anomaly detection methods to automatically suppress the background, which has the advantages of few hyperparameters and strong generalization performance. Attached Figure Description

[0075] Figure 1 This is a flowchart illustrating the method for detecting small moving targets in a high-altitude, top-down scenario according to the present invention.

[0076] Figure 2This is a schematic diagram of the composition of the moving small target detection device in a high-altitude top-down scene described in this invention. Detailed Implementation

[0077] The present invention will now be described in detail with reference to the accompanying drawings and embodiments:

[0078] like Figure 1 As shown, the method for detecting small moving targets in a high-altitude overhead view scenario according to the present invention includes the following steps in sequence:

[0079] A: Acquire continuous images of a high-altitude, top-down scene, and sequentially use each frame of the continuous images as an intermediate frame. Combine the intermediate frame and the T frames before and after it to form an image sequence a to be recognized. i Finally, a set of image sequences to be identified is obtained, A = {a1, a2, ..., a...} N}, i = 1, 2, ..., N, where N is the total number of frames in the continuous images;

[0080] In this invention, continuous images of a high-altitude, top-down scene can be acquired using an image acquisition device. When the i-th frame is used as an intermediate frame, the i-th frame and the T frames preceding and following it, i.e., the iT-th to i+T-th frames, are combined to form the image sequence a to be identified. i Finally, the set of image sequences to be identified is obtained as A = {a1, a2, ..., a...} N}, i = 1, 2, ..., N;

[0081] In this embodiment, in order to obtain more accurate image data and further improve the detection accuracy, if there are less than T frames before the i-th frame, then frames are taken sequentially from the first frame before the i-th frame to the 2T+1-th frame; if there are less than T frames after the i-th frame, then frames are taken sequentially from the last frame after the i-th frame to the 2T+1-th frame. For example, when T=2, for the 1st frame (i=1), the 1st to 5th frames are combined to form the image sequence to be identified, a1; for the 2nd frame (i=2), the 1st to 5th frames are combined to form the image sequence to be identified, a2; similarly, for the second-to-last frame (i=N-1), the last frame to the fifth-to-last frame are combined to form the image sequence to be identified; for the last frame (i=N), the last frame to the fifth-to-last frame are combined to form the image sequence to be identified.

[0082] B: Construct a visual perception module and use the visual perception module to identify the image sequence a. i Gaussian smoothing is applied to the pixel values ​​of each pixel in each frame of the image to obtain a smoothed feature map with precise location information for each frame. This smoothed feature map is then used to identify the image sequence a. i The smoothed feature maps corresponding to each frame of the image constitute a smoothed feature map sequence A.p ;

[0083] Because the receptive field shape of retinal neurons in a biological visual system approximates a Gaussian distribution when receiving external images, the biological retinal neurons essentially perform Gaussian filtering on the input image. In this invention, the visual perception module uses a two-dimensional Gaussian filter to simulate the function of retinal neurons, smoothing the pixel value of each pixel in each frame of the input image to facilitate subsequent modules in detecting small moving targets.

[0084] Let a be the image sequence to be identified for moving small target detection. i The duration is 2T+1 frames, for the image sequence to be identified a i Input any image in the dataset. After passing through the visual perception module, a smooth feature map P(x,y,t) with precise location information is output.

[0085]

[0086] Where x and y represent the spatial horizontal and vertical coordinates of a pixel, respectively, and t represents time, which also represents the current frame image in the image sequence to be identified, a. i The order of the items, Let represent the set of real numbers, u represent the integration variable corresponding to x, and v represent the integration variable corresponding to y. This represents the first Gaussian kernel function. σ1 represents the standard deviation of the first Gaussian kernel function;

[0087] Finally, using the image sequence to be identified, a i The 2T+1 smoothed feature maps obtained after Gaussian smoothing of all images form a smoothed feature map sequence A. p ;

[0088] C: Construct a change-aware module to process the smooth feature map sequence A. p Spatial and temporal features are extracted from the smoothed feature map to obtain the smoothed feature map sequence A. p intermediate frame N p The corresponding first spatiotemporal dynamic feature map;

[0089] In this invention, a change-sensing module is constructed to process the smooth feature map sequence A. p intermediate frame N p Spatial feature extraction is performed, and the smoothed feature map sequence A is... p Except for intermediate frame N p The temporal information of the remaining frames is integrated into the intermediate frame N. p Temporal feature extraction is performed, ultimately yielding a smooth feature map sequence A. p Inner intermediate frame Np The corresponding first spatiotemporal dynamic feature map;

[0090] In the avian visual pathway, there is extensive overlap between visual neural information transmitted from the retina and the dendritic terminals of OTid neurons (OT neurons for short). Each individual brush dendrite of an OT neuron can aggregate and connect with multiple retinal ganglion cells.

[0091] Let F be the receptive field of a dendrite in an OT neuron. Within receptive field F, multiple points receiving input from the previous layer, each with precise spatiotemporal location information, are considered. Because the dendrites of an OT neuron have small, densely distributed receptive fields for the input, this structure makes it highly sensitive to changes in input intensity.

[0092] To simulate the change-sensitive characteristic of OT neuron dendrites, this invention employs a change-sensing module to perceive changes in small targets across spatial and temporal dimensions. This change-sensing module consists of m multi-channel spatiotemporal three-dimensional filter banks with different preference directions, used to calculate intermediate frames N. p The change in pixel value of each pixel in the corresponding smooth feature map in the temporal and spatial dimensions.

[0093] In this embodiment, let Q represent m multi-channel spatiotemporal three-dimensional filter banks in the change sensing module, and let F be one of the multi-channel spatiotemporal three-dimensional filter banks. j The value of j ranges from {1,2,…,m}, and the intermediate frame N is calculated through convolution. p The response value of each pixel on the corresponding smooth feature map P1(x,y,t) in the preference direction θ is used to obtain the intermediate frame N. p The corresponding first spatiotemporal dynamic feature map

[0094]

[0095]

[0096] in, This represents the first spatiotemporal dynamic feature map. Represents a spatiotemporal three-dimensional filter;

[0097] In this invention, each preference direction θ corresponds to a And as a multi-channel spatiotemporal three-dimensional filter bank F j A channel, multiple preference directions θ corresponding to multiple Together they form the corresponding multi-channel spatiotemporal three-dimensional filter bank F j ; The phase offset of the spatiotemporal three-dimensional filter is given by F in a multi-channel spatiotemporal three-dimensional filter bank. j middle, For a given value, m have different values. Multi-channel spatiotemporal three-dimensional filter bank F j This constitutes Q. For matrix multiplication calculation, It is a two-dimensional Gabor filter, representing a three-dimensional spatiotemporal filter. For the two-dimensional spatial part of P1(x,y,t), perform a convolution operation on the spatial part P1(x,y).

[0098] x'(θ) = xcosθ + ysinθ;

[0099] y'(θ) = -xsinθ + ycosθ;

[0100]

[0101] Where x′(θ) and y′(θ) are the coordinates of x and y in the preference direction θ, respectively, and γ, σ², and λ represent the two-dimensional Gabor filter. The spatial aspect ratio, standard deviation, and wavelength are constants in this example;

[0102] In this invention, i(t) is the temporal kernel function for detecting pixel changes, and the time dimension of P1(x,y,t) is the value of pixel (x,y) at different times t in space. Since the input in this invention is an ordered image sequence, one time unit corresponds to one frame in the image sequence. A convolution operation is performed on the temporal part P1(t) of P1(x,y,t) to calculate the intermediate frame N. p The value of each pixel changes over 2T+1 frames (T frames before and T frames after), and the temporal changes are extracted using a first-order differential operator. Smooth feature map sequence A p Except for intermediate frame N p Information from the pixels in the remaining frames is integrated into the intermediate frame N using a first-order differential operator. p The Sobel operator is a commonly used first-order differential operator. This invention uses the Sobel operator in the time dimension to calculate the change of each point within a time window, achieving the effects of obtaining time-domain changes and Gaussian smoothing.

[0103] By following the steps above, the smoothed feature map sequence A output from step B can be obtained. p The spatiotemporal information contained therein is filtered through a three-dimensional spatiotemporal filter. Integrate into the Nth p In the frame feature map, spatiotemporal dynamic features are extracted, ultimately yielding the intermediate frame N. p The corresponding first spatiotemporal dynamic feature map Due to the convolution operation, the first spatiotemporal dynamic feature map obtained in step C can sensitively capture the changes in pixel values ​​in various directions within the receptive field F, and can effectively extract motion information.

[0104] D: Construct a motion information calculation module, and use the motion information calculation module to perform motion energy calculation, receptive field enlargement, motion target scale screening and background suppression on the first spatiotemporal dynamic feature map obtained in step C, to obtain an optimized second spatiotemporal dynamic feature map;

[0105] Since OT neuron dendrites only sense changes, but merely recognizing changes in point intensity is insufficient to identify moving targets, flashing points will also output a strong response in the aforementioned calculations. This invention, through research on the transmission process of motion information in the avian visual pathway, concludes that an OT neuron tree can activate cell bodies, and the output of an OT neuron tree only contains target-related information. Therefore, the OT neuron cell body, as an important computational unit in the pathway, possesses the following characteristics:

[0106] 1. Motion speed and direction are synthesized in the OT cell body;

[0107] 2. Small-scale inputs can activate the next layer of neurons, therefore the receptive field of OT neurons increases through max pooling;

[0108] 3. The OT cell body can calculate the relative motion between the background and the target based on the background intensity change information received by the dendrites, and small targets can be screened at the cell body layer.

[0109] Based on the characteristics of OT neuron cell bodies obtained from the above analysis, the motion information calculation module in this invention is divided into a motion energy calculation submodule, a receptive field enlargement submodule, a motion target scale screening submodule, and a background suppression submodule. The above submodules are used sequentially to perform motion energy calculation, receptive field enlargement, motion target scale screening, and background suppression on the first spatiotemporal dynamic feature map obtained in step C, and finally obtain the optimized second spatiotemporal dynamic feature map.

[0110] Step D includes the following specific steps:

[0111] D1: Construct a motion energy calculation submodule and use the motion energy calculation submodule to calculate the net motion energy feature map of each pixel in the first spatiotemporal dynamic feature map in each preferred direction;

[0112] Step D1 includes the following specific steps:

[0113] D11: Due to the multi-channel spatiotemporal three-dimensional filter bank F in step C j Each channel in the process calculates a first spatiotemporal dynamic feature map, and the phase shift difference in the m-channel spatiotemporal three-dimensional filter bank Q is... The two multi-channel spatiotemporal three-dimensional filter banks are grouped together and designated as F. g and F h , will F g and F h Filters with the same θ (i.e., the same channel) are considered as a set of orthogonal filters, and the two first spatiotemporal dynamic feature maps generated accordingly are considered as a set of orthogonal first spatiotemporal dynamic feature maps; the first motion energy feature map E is calculated from the set of orthogonal first spatiotemporal dynamic feature maps. θ (x,y,t) is the square root of the sum of squares of a set of orthogonal first-order spatiotemporal dynamic feature maps:

[0114]

[0115] D12: Based on the first motion energy feature map calculated in step D11, calculate the net motion energy feature map of each pixel in the first motion energy feature map in each preferred direction using the following formula:

[0116]

[0117] in, Represents the net kinetic energy characteristic diagram, E flk Flicker Energy is defined as the average value of the output energy. M represents the number of possible values ​​for the preference direction λ. In this embodiment, the range of values ​​for θ is... That is, M=8; when calculating the net kinetic energy characteristic diagram, if If the value exceeds B, then take b1 = 2π - b0; for example, when hour, If it is not in the range of values ​​B, then take... b1 can be used if it is within the range B. The corresponding net kinetic energy characteristic map is calculated.

[0118] D2: Construct a receptive field enlargement submodule and use it to modify the net kinetic energy feature map. The receptive field is enlarged, and the second motion energy feature map E′ after the receptive field enlargement is finally obtained. θ (x,y,t);

[0119] In this invention, the receptive field enlargement submodule simulates the receptive field enlargement process of OT neurons in response to input through pooling. Pooling layers reduce model size, increase computational speed, and enhance the invariance of model response to features. In this embodiment, for Use the max pooling method Enlarge the receptive field, that is:

[0120]

[0121] Where k1 is the size of the first pooling window; during pooling, a fixed window is taken and slides across all regions of the input net motion energy feature map according to the set stride size, and the maximum value of all pixels in the pooling window is calculated, finally obtaining the second motion energy feature map E′ after the receptive field is enlarged. θ (x,y,t).

[0122] D3: Construct a motion target scale filtering submodule, and use the motion target scale filtering submodule to perform scale filtering on the motion features in the second motion energy feature map to obtain the scale-filtered third motion energy feature map;

[0123] In this invention, the enlarged receptive field E′ obtained in step D2 θ (x,y,t) can detect the motion of targets at various scales. Since this application needs to extract small moving targets, further scale filtering is required to extract small moving targets.

[0124] In this embodiment, a scale-selection kernel function considering a spatial domain is constructed, and scale selection is performed through convolution calculation to obtain the scale-selected third motion energy feature map; that is:

[0125] S θ (x,y,t)=[∫∫E′ θ (u,v,t)W s (xu,yv)dudv];

[0126] Among them, S θ (x,y,t) is the third motion energy characteristic map, W s (·) denotes the lateral inhibition scale-selective kernel function, [ ] + It is a linear rectified function;

[0127] In this embodiment, the kernel function W for lateral suppression scale selection is... s (·) can be represented in space as a receptive field, with the response zone in the middle (the input weights within this zone are positive); when the scale of the object's motion energy is smaller than the response zone, the kernel function responds strongly to the pixels within the response zone, and the smaller the target, the larger the weight of the kernel function; when it is larger than the response zone, the neuron's response will be suppressed. + As a linear rectified function, if the target is larger than the excitation region, it is strongly suppressed by the kernel function suppression region (with negative weights), and the output is close to 0 or negative, thus achieving suppression of large-scale targets and large-area low-frequency backgrounds, thereby achieving scale selectivity.

[0128] In this embodiment, the side suppression scale selection kernel function that satisfies the above conditions is:

[0129]

[0130] Where a is the scale adjustment parameter and μ is a constant used to adjust the degree of suppression;

[0131] D4: Construct a background suppression submodule and use it to suppress the background of the third motion energy feature map; finally, the output map after background suppression is obtained, which is the optimized second spatiotemporal dynamic feature map O. θ (x,y,t);

[0132] In aerial scenes, birds appear to have a background that moves in the same direction. This means that time-series data collected from aerial views has a priori conditions: the number of background pixels is much greater than the number of pixels occupied by small objects, and the background moves in the same direction. In other words, in the image sequence a to be identified... i In the context, background pixels that are not the target have similar motion energy in direction and magnitude.

[0133] In biological visual systems, due to lateral inhibition, a neuron tuned to a particular velocity and direction may be inhibited by other neurons tuned to nearby velocities and directions. Therefore, inhibition is strongest when stimuli surrounding a point have the same direction and velocity of motion as stimuli at the relevant point, and least when surrounding stimuli move in the opposite direction to the central stimulus.

[0134] In order to make the constructed background suppression submodule consistent with the conclusions of neurophysiology and to suppress a wide range of inputs with the same velocity characteristics, the background suppression submodule in this invention models the data of the third motion energy feature map, transforming the distinction between background and target into an anomaly detection problem. Values ​​in the background model that do not conform to the model distribution are regarded as outliers and as potential locations of the target, while all non-outliers are suppressed.

[0135] Anomaly detection is one of the core problems in data mining. Commonly used unsupervised anomaly detection methods include statistical methods, clustering, and isolated forest methods. This invention uses statistical methods for anomaly detection. In statistics, the normality of data is assumed to follow a statistical model, and outliers are abnormal observations that are far removed from other values. This invention uses the calculation of the Z-score of observation points as an example to describe the background suppression method based on the statistical model:

[0136] The third motion energy characteristic diagram S in any direction θ (x,y,t), calculate the standard deviation σ of its distribution. f ;

[0137]

[0138] Among them, the image sequence to be identified, a i The size of any image in the dataset is H×W. Since the image size is not changed in steps A to D, the third motion energy feature map S... θ The dimensions of (x,y,t) are also H×W, and mean(·) is calculated by taking the average value. Then the Z-score of the corresponding pixel (x,y,t) is:

[0139]

[0140] The Z-score threshold is set to ε, in this embodiment ε = 3*σ f Pixels with a Z-score threshold ε or less are considered to conform to the background model distribution, while pixels with a Z-score threshold ε are considered outliers, i.e., target points. The final output image after background suppression is the optimized second spatiotemporal dynamic feature map O. θ (x,y,t) is:

[0141] O θ (x, y, t) = S θ (x, y, t) × [Zscore] θ (x,y,t)-ε] + ;

[0142] Through the above steps, the motion energy calculation, receptive field enlargement, moving target scale screening, and background suppression operations of the first spatiotemporal dynamic feature map can be completed, and the optimized second spatiotemporal dynamic feature map can be obtained.

[0143] E: Construct a receptive field enlargement module, and use the receptive field enlargement module to enlarge the receptive field of the second spatiotemporal dynamic feature map obtained in step D, and finally obtain the third spatiotemporal dynamic feature map;

[0144] The round nucleus (Rt) is the largest single nucleus in the thalamus of most birds. The Rt has several anatomical branches, each receiving projections from OT neurons. A notable feature of OT-Rt projection is the complete loss of one-to-one correspondence. That is, the large receptive field of OT neurons coarsens the precise spatial map of retinal input at the OT level, resulting in a second enlargement of the receptive field in the projections of OT neurons onto the Rt.

[0145] Analysis revealed that the receptive fields of most round nucleus neurons in birds almost cover the entire visual field; therefore, this invention utilizes the max pooling method. The process of simulating the receptive field increasing again is used to obtain the third spatiotemporal dynamic feature map O′. θ (x,y,t);

[0146]

[0147] in, For max pooling, k2 is the size of the second pooling window;

[0148] F: Following the methods in steps B to E, sequentially identify each image sequence a in the image sequence set A. i The process is performed to obtain the corresponding third spatiotemporal dynamic feature maps, and the position where the pixel value in the third spatiotemporal dynamic feature map reaches the maximum value is taken as the position of the moving small target.

[0149] In this invention, the position of the moving small target is the location where the pixel value in the third spatiotemporal dynamic feature map reaches its maximum value. Since different pixels have different preferred motion directions θ, the corresponding third spatiotemporal dynamic feature map is calculated for each pixel in each frame image according to its different preferred motion directions θ. In this embodiment, a total of 8 preferred directions are set. The position of the pixel value at the maximum value after summing the third spatiotemporal dynamic feature maps corresponding to all preferred directions is the location of the moving small target in the current frame (x). s ,y s );

[0150]

[0151] Where, x s y s , respectively, are the x and y coordinates of the moving small target detected at time t.

[0152] like Figure 2 As shown, the moving small target detection device in a high-altitude overhead view scenario according to the present invention includes an image acquisition module, a visual perception module, a change perception module, a motion information calculation module, a receptive field enlargement module, and a target position recognition module, wherein:

[0153] The image acquisition module is used to acquire continuous images of a high-altitude overhead scene according to the method described in step A, and generate a set A of image sequences to be identified, A = {a1, a2, ..., a...} N}, i = 1, 2, ..., N, where N is the total number of frames in the continuous images;

[0154] The visual perception module is used to acquire the image sequence a to be recognized according to the method described in step B. i Each frame of the image contains a smoothed feature map with precise location information, and the image sequence to be identified is a... i The smoothed feature maps corresponding to each frame of the image constitute a smoothed feature map sequence A. p ;

[0155] The change-aware module is used to process the smoothed feature map sequence A according to the method described in step C. p Spatial and temporal features are extracted from the smoothed feature map to obtain the smoothed feature map sequence A. p intermediate frame N p The corresponding first spatiotemporal dynamic feature map;

[0156] The motion information calculation module is used to perform motion energy calculation, receptive field enlargement, target scale screening and background suppression sequentially on the first spatiotemporal dynamic feature map according to the method described in step D, so as to obtain the optimized second spatiotemporal dynamic feature map.

[0157] The motion information calculation module includes a motion energy calculation submodule, a receptive field enlargement submodule, a motion target scale filtering submodule, and a background suppression submodule.

[0158] The motion energy calculation submodule is used to calculate the net motion energy feature map of each pixel in the first spatiotemporal dynamic feature map in each preferred direction, according to the method described in step D1.

[0159] The receptive field enlargement submodule is used to enlarge the receptive field of the net motion energy feature map according to the method described in step D2, so as to obtain a second motion energy feature map with enlarged receptive field.

[0160] The motion target scale filtering submodule is used to perform scale filtering on the motion features in the second motion energy feature map according to the method described in step D3, so as to obtain the scale-filtered third motion energy feature map.

[0161] The background suppression submodule is used to suppress the background of the third motion energy feature map according to the method described in step D4; the final output map after background suppression is the optimized second spatiotemporal dynamic feature map.

[0162] The receptive field enlargement module is used to enlarge the receptive field of the second spatiotemporal dynamic feature map a second time according to the method in step E, and finally obtain the third spatiotemporal dynamic feature map.

[0163] The location recognition module is used to sequentially obtain each image sequence a in the image sequence set A to be recognized, according to the method in step F. i The corresponding third spatiotemporal dynamic feature map is used, and the position where the pixel value in the third spatiotemporal dynamic feature map reaches the maximum value is taken as the position of the moving small target.

Claims

1. A method for detecting small moving targets in a high-altitude overhead view scenario, characterized in that: Includes the following steps: A: Acquire continuous images of a high-altitude, top-down scene, and sequentially use each frame of the continuous images as an intermediate frame. Then, combine the intermediate frames and the frames before and after them... The frames of images form a sequence of images to be identified. Finally, a set of image sequences to be identified is obtained. , The total number of frames in a continuous image; B: Construct a visual perception module and use the visual perception module to identify the image sequence. Gaussian smoothing is applied to the pixel values ​​of each pixel in each frame of the image to obtain a smoothed feature map with precise location information for each frame. This smoothed feature map is then used to identify the image sequence. The smoothed feature maps corresponding to each frame of the image constitute a smoothed feature map sequence. ; C: Construct a change-aware module to process smooth feature map sequences. Spatial and temporal features are extracted from the smoothed feature maps to obtain a smoothed feature map sequence. intermediate frames in The corresponding first spatiotemporal dynamic feature map; D: Construct a motion information calculation module, and use the motion information calculation module to perform motion energy calculation, receptive field enlargement, motion target scale screening and background suppression on the first spatiotemporal dynamic feature map obtained in step C, to obtain an optimized second spatiotemporal dynamic feature map; E: Construct a receptive field enlargement module, and use the receptive field enlargement module to enlarge the receptive field of the second spatiotemporal dynamic feature map obtained in step D, and finally obtain the third spatiotemporal dynamic feature map; F: According to the method in steps B to E, sequentially process each to-be-recognized image sequence in the to-be-recognized image sequence set to obtain a corresponding third spatiotemporal dynamic feature map, and take the position where the pixel point value in the third spatiotemporal dynamic feature map is maximum as the position of the moving small target. ​ 2. The method for detecting moving small targets in a high-altitude overhead view scene according to claim 1, characterized in that: In step B, the to-be-identified image sequence to be subjected to the motion small target detection has a time length of frames, for any image in the to-be-identified image sequence , input is subjected to the visual perception module, and a smooth feature map with accurate position information is output ; in, These represent the horizontal and vertical coordinates of a pixel, respectively. Indicates time, Represents the set of real numbers. express The corresponding integral variable, express The corresponding integral variable, This represents the first Gaussian kernel function. , This represents the standard deviation of the first Gaussian kernel function; Finally, a plurality of smoothed feature maps are obtained by Gaussian smoothing all images in the image sequence to be recognized The plurality of smoothed feature maps constitute a smoothed feature map sequence .​ 3. The method for detecting moving small targets in a high-altitude overhead view scene according to claim 1, characterized in that: In step C, the change-sensing module consists of m multi-channel spatiotemporal three-dimensional filter banks with different preference directions; through the change-sensing module, the smooth feature map sequence is processed. intermediate frames in Spatial feature extraction is performed, and the smoothed feature map sequence is then processed. Except for intermediate frames Temporal information from all other frames is integrated into the intermediate frame. Temporal feature extraction is performed to obtain a smooth feature map sequence. Inner frame The corresponding first spatiotemporal dynamic feature map.

4. The method for detecting moving small targets in a high-altitude overhead view scene according to claim 3, characterized in that: In step C, let Indicating change perception module A multi-channel spatiotemporal three-dimensional filter bank, one of which is represented as: , The range of values ​​is Calculate intermediate frames through convolution The corresponding smooth feature map Each pixel in the preferred direction The response value on the screen is used to obtain the intermediate frame. The corresponding first spatiotemporal dynamic feature map ; in, This represents the first spatiotemporal dynamic feature map. Represents a spatiotemporal three-dimensional filter; Each preference direction Each corresponds to one Furthermore, it serves as a multi-channel spatiotemporal three-dimensional filter bank. One channel, multiple preference directions The corresponding multiple Together they form the corresponding multi-channel spatiotemporal three-dimensional filter bank ; The phase offset of a spatiotemporal three-dimensional filter in a multi-channel spatiotemporal three-dimensional filter bank. middle, For a constant value, Each has different Multi-channel spatiotemporal three-dimensional filter bank It constitutes , For matrix multiplication calculation, It is a two-dimensional Gabor filter, representing a three-dimensional spatiotemporal filter. The two-dimensional part of space, for airspace portion Perform convolution operations: in, and In the direction of preference superior The corresponding coordinates , and These represent two-dimensional Gabor filters. Spatial aspect ratio, standard deviation, and wavelength A temporal kernel function for detecting pixel changes. .

5. The method for detecting moving small targets in a high-altitude overhead view scenario according to claim 1, characterized in that: Step D includes the following specific steps: D1: Construct a motion energy calculation submodule and use the motion energy calculation submodule to calculate the net motion energy feature map of each pixel in the first spatiotemporal dynamic feature map in each preferred direction; D2: Construct a receptive field enlargement submodule and use it to modify the net kinetic energy feature map. The receptive field is enlarged to obtain the second motion energy feature map after the receptive field enlargement. ; ； in, The size of the first pooling window. This is a max pooling operation; D3: Construct a motion target scale filtering submodule, and use the motion target scale filtering submodule to perform scale filtering on the motion features in the second motion energy feature map to obtain the scale-filtered third motion energy feature map; in, This is the third kinetic energy characteristic diagram. This indicates the kernel function for selecting the side suppression scale. It is a linear rectified function. express The corresponding integral variable, express The corresponding integral variable; D4: Construct a background suppression submodule and use it to suppress the background of the third motion energy feature map; Third kinetic energy characteristic diagram in any direction Calculate the standard deviation of its distribution. ; Among them, the third motion energy characteristic diagram The size is , To calculate the average value, the corresponding pixel point The Z-score is: Set the Z-score threshold to ,in Less than or equal to the threshold The pixels are considered to follow the background model distribution and are greater than the Z-score threshold. Pixels that are not properly identified are considered outliers, i.e., target points. The final output image after background suppression is the optimized second spatiotemporal dynamic feature map. for:

6. The method for detecting small moving targets in a high-altitude overhead view scene according to claim 5, characterized in that: Step D1 includes the following specific steps: D11: Will A multi-channel spatiotemporal three-dimensional filter bank The phase offset difference is The two multi-channel spatiotemporal three-dimensional filter banks are grouped together and set as follows: and ,Will and Middle preference direction The same filter is considered as a set of orthogonal filters, and the two corresponding first spatiotemporal dynamic feature maps are considered as a set of orthogonal first spatiotemporal dynamic feature maps; the first motion energy feature map is calculated from the set of orthogonal first spatiotemporal dynamic feature maps. The square root of the sum of squares of a set of orthogonal first-order spatiotemporal dynamic feature maps: D12: Based on the first motion energy feature map calculated in step D11, calculate the net motion energy feature map of each pixel in the first motion energy feature map in each preferred direction using the following formula: in, This represents a graph depicting the characteristics of net kinetic energy. The flicker energy is represented by the average value of the output energy. , For preferred direction The number of possible values.

7. The method for detecting small moving targets in a high-altitude overhead view scene according to claim 1, characterized in that: In step E, the max pooling method is used. The process of simulating the receptive field increasing again is used to obtain the third spatiotemporal dynamic feature map. ; in, For max pooling operation, This is the size of the second pooling window.

8. The method for detecting moving small targets in a high-altitude overhead view scene according to claim 1, characterized in that: In step F, different preference directions of pixels in each frame of the image are considered. Calculate the corresponding third spatiotemporal dynamic feature map for each preference direction, and then sum the third spatiotemporal dynamic feature maps for all preferences. The position of the maximum pixel value is the location of the moving small target in the current frame. ; in, They are respectively in The horizontal and vertical coordinates of the moving small target detected at any time.

9. A device for detecting small moving targets in a high-altitude overhead view scenario based on the detection method of any one of claims 1 to 8, characterized in that: It includes an image acquisition module, a visual perception module, a change perception module, a motion information calculation module, a receptive field enlargement module, and a target location recognition module. The image acquisition module is used to acquire continuous images of a high-altitude, top-down scene and generate a set of image sequences to be identified. , The total number of frames in a continuous image; The visual perception module is used to acquire the image sequence to be recognized. Each frame of the image corresponds to a smooth feature map with precise location information, and the image sequence to be identified is used as the basis for this process. The smoothed feature maps corresponding to each frame of the image constitute a smoothed feature map sequence. ; The change-aware module is used to process smooth feature map sequences. Spatial and temporal features are extracted from the smoothed feature maps to obtain a smoothed feature map sequence. intermediate frames in The corresponding first spatiotemporal dynamic feature map; The motion information calculation module is used to sequentially perform motion energy calculation, receptive field enlargement, target scale screening and background suppression on the first spatiotemporal dynamic feature map to obtain the optimized second spatiotemporal dynamic feature map. The receptive field enlargement module is used to enlarge the receptive field of the second spatiotemporal dynamic feature map a second time, and finally obtain the third spatiotemporal dynamic feature map. The location recognition module is used to sequentially acquire a set of image sequences to be recognized. Each image sequence to be identified The corresponding third spatiotemporal dynamic feature map is used, and the position where the pixel value in the third spatiotemporal dynamic feature map reaches the maximum value is taken as the position of the moving small target.

10. The device for detecting moving small targets in a high-altitude overhead view scene according to claim 9, characterized in that: The motion information calculation module includes a motion energy calculation submodule, a receptive field enlargement submodule, a motion target scale filtering submodule, and a background suppression submodule; The motion energy calculation submodule is used to calculate the net motion energy feature map of each pixel in the first spatiotemporal dynamic feature map in each preferred direction; The receptive field enlargement submodule is used to enlarge the receptive field of the net motion energy feature map to obtain a second motion energy feature map with enlarged receptive field. The motion target scale filtering submodule is used to perform scale filtering on the motion features in the second motion energy feature map to obtain the scale-filtered third motion energy feature map. The background suppression submodule is used to suppress the background of the third motion energy feature map; The final output map after background suppression is the optimized second spatiotemporal dynamic feature map.

Images