A method for aerial target salience detection based on eagle-eye vision and double fovea

Abstract

This invention discloses an aerial target salience detection method based on eagle-eye vision with dual fovea, comprising: 1) modeling an eagle-eye vision fovea-like mechanism; 2) modeling an eagle-eye vision lateral fovea-like mechanism; 3) modeling an eagle-eye four-color mechanism; 4) fusion of eagle-eye dual fovea; and 5) outputting a salience map of aerial targets based on eagle-eye vision with dual fovea. This invention introduces raptor vision mechanisms into the aerial target salience detection process, better mapping the raptor's ability to capture mid-to-long-range prey to the task requirements of UAVs for mid-to-long-range target identification. It also combines deep learning methods with traditional visual methods, avoiding the need for large datasets with pixel annotations while still achieving good target salience results. Finally, it improves the eagle-eye four-color mechanism by fusing salience values ​​from different channels, avoiding focusing on a single channel, and utilizing spatial information to correct color detection results, thus improving salience performance.

Description

Technical Field

[0001] This invention provides a method for detecting the salience of aerial targets using a dual-fovea approach similar to eagle-eye vision, belonging to the field of computer vision positioning and recognition technology. Background Technology

[0002] Compared to rotary-wing aircraft, fixed-wing drones have advantages such as longer flight time, higher flight speed, and higher flight altitude, and are therefore widely used in various fields such as power line inspection, geological monitoring, and communication relay.

[0003] During drone missions, target detection can be achieved through various methods, such as airborne electro-optical guidance, global navigation modules, and visual positioning. However, airborne electro-optical guidance systems are expensive and require sophisticated instruments; each impact from the drone can cause damage to the system, making them unsuitable for long-duration, multi-mission missions. Global navigation modules are susceptible to interference from external electromagnetic signals, resulting in low detection accuracy. Visual positioning, on the other hand, only requires inexpensive industrial cameras to guide the drone, is unaffected by the environment, and has a wider range of applications. However, aerial targets detected by a drone at long distances can be considered small targets within the drone's field of view, resulting in low image resolution and limited information. Therefore, the drone needs to accurately pinpoint the target's exact location at long distances to prepare for subsequent missions.

[0004] The visual systems of birds of prey such as eagles are characterized by a large field of vision, high sensitivity, high recognition accuracy, and long recognition distance. This is because the eyes and visual processing organs of birds of prey possess unique physiological structures and functional characteristics. The eyes of eagles are arranged in a binocular, laterally distributed structure, which gives them a large field of vision. Furthermore, the double foveal structure, such as... Figure 1 As shown, there are numerous photoreceptor cells in these two fovea regions. The visual sensitivity of an eagle's eye is directly proportional to the density of photoreceptors. The distribution of photoreceptor cells in the eagle's retina is non-uniform, with the highest density in the central fovea and followed by the lateral fovea. Due to this structure, these two high-resolution imaging regions image objects at different distances. Eagles use the monocular central fovea to observe prey at long distances and the binocular lateral fovea to track and capture prey at close distances. Photoreceptors are generally divided into cone cells and rod cells. Cone cells, due to their visual pigments and oil droplets, can sense light intensity and color, while rod cells are mainly used for shape and motion perception. This distribution makes the visual sensitivity of an eagle's eye higher than that of other organisms. Furthermore, research has found that animals have a different color perception system than humans. Among the photoreceptor cells in an eagle's eye are four types of single cone cells, each of which can be further divided into violet-sensitive, short-wavelength-sensitive, medium-wavelength-sensitive, and long-wavelength-sensitive types based on their sensitive wavelength. The physiological structure of a single cone cell is as follows: Figure 2As shown, different cone cells will form four antagonistic receptor pathways after receiving the same stimulus. This color method can make each channel retain the color information of its corresponding wavelength. When an eagle is flying high in the sky, if it knows which wavelength of light the prey is in, it can focus its attention on the corresponding channel, which greatly reduces the detection area.

[0005] Therefore, how to learn from and imitate the visual systems of birds of prey such as eagles to provide high-precision aerial target localization and salience detection for drones has become an urgent problem to be solved. Summary of the Invention

[0006] This invention addresses the problem of identifying small aerial targets at medium to long distances in complex environments using unmanned aerial vehicles (UAVs). It proposes an aerial target salience detection method based on eagle-eye vision and dual fovea, which effectively improves the detection efficiency and accuracy of targets.

[0007] Technical solution of the present invention

[0008] A method for detecting the salience of aerial targets using a bifoveal model similar to eagle-eye vision includes the following steps:

[0009] Step 1: Modeling the concave mechanism at the center of the eagle eye's vision;

[0010] Step 2: Modeling the concave-lateral mechanism of eagle-eye vision;

[0011] Step 3: Modeling based on the Eagle Eye four-color mechanism;

[0012] Step 4: Fusion of the eagle-eye double concave shape;

[0013] Step 5: Output a salient image of aerial targets with dual fovea, mimicking eagle-eye vision.

[0014] The above plan further includes:

[0015] Step 1: Modeling the concave mechanism of eagle eye vision

[0016] A network model of the central concave shape of eagle eye vision is established. The backbone of the network is a 50-layer residual network. After the input image passes through the network, the predicted target location and type are obtained, as well as the five-layer feature map information of forward propagation. The image within the predicted location and the predicted type are used as the input for step two.

[0017] Step 2: Modeling the foveal mechanism of eagle eye vision

[0018] After upsampling the target region, the image is processed by Gaussian filtering to retain low-frequency information. Finally, the color channels of the Gaussian-filtered image are separated.

[0019] Step 3: Modeling based on the eagle-eye four-color mechanism

[0020] By analyzing the mutual antagonism of different cone cells in the eagle eye after processing the same signal, four receptor pathways were obtained. Each pathway represents the contrast response of the corresponding cone cell with other types of cone cells. A perception model of the eagle eye four-color mechanism was constructed based on this contrast response. The distance between the signal pixel values ​​of each pathway was calculated, and the histogram of the output image of each pathway was statistically analyzed. The color saliency detection results of the eagle eye four-color mechanism were calculated.

[0021] Step 4: Fusion of the eagle-eye double concave shape

[0022] Upsampling is performed using the multi-layer feature map information obtained from the network forward propagation in step one to calculate spatial information. Then, the spatial information is fused with the color saliency detection result from the previous step to obtain a fused saliency output.

[0023] Step 5: Output a salience map of aerial targets using a dual-fovea model similar to eagle-eye vision.

[0024] The obtained saliency outputs are returned to the large field of view according to their corresponding positional relationships. The erosion-fill operation is then used to remove discontinuous small regions in the image to obtain the final saliency detection result.

[0025] Furthermore, the modeling of the eagle-eye visual concave mechanism in step one includes:

[0026] An EagleFoveaNet network, mimicking the fovea of ​​an eagle's eye, was constructed. The backbone of the network was a 50-layer ResNet50 residual network. Based on the variable spatial resolution formed by the focal points of the two fovea regions in an eagle's eye, a four-layer feature map pyramid network structure was established to detect targets of different sizes. The loss function L was defined as:

[0027] L=∑L cls +λ1ΣL reg +λ2ΣL sal

[0028] Where λ1 and λ2 are constants, L cls L is the classification loss predicted by the network. reg L is the regression loss for predicting bounding boxes in the network. sal Predict the salient loss within the bounding box for the network;

[0029] Let the significance value at target i be I, and the significance value of the currently predicted candidate box j be I′. Then the significance loss here is defined as:

[0030]

[0031] Where i and j represent the set of target pixels and the set of candidate bounding box pixels, respectively. This represents the overlapping area between the target and the candidate box, where ||·|| is the difference in significance values ​​at the same location.

[0032] After the input image passes through the EagleFoveaNet network, the predicted target location and category are obtained, along with the five-layer feature map information from the forward propagation. The image within the predicted location and the predicted category are then used as input for step two.

[0033] Furthermore, the modeling of the foveal mechanism mimicking eagle-eye vision in step two includes:

[0034] By using a telephoto lens to simulate the detailed features of objects observed by an eagle at close range, that is, to simulate the small scene of the fovea on the visual side of the eagle's eye, and to simulate the fovea imaging on the visual side of the eagle's eye by using a bicubic interpolation upsampling algorithm.

[0035] Let image1 be the image within the predicted location, and image2 be the magnified image. Let F′(x,y) be the pixel value corresponding to the previous pixel in image2, where x and y are the coordinates on the image. Then, the value of F′(x,y) calculated by bitrilinear interpolation is:

[0036]

[0037] Where W(i) and W(j) are the weight coefficients of each point, and F(i,j) are the pixel values ​​of x and y mapped back to the corresponding 4×4 pixel boxes near the position in image1;

[0038] After upsampling the target region, Gaussian filtering is performed on image2 to remove high-frequency components and retain low-frequency information.

[0039] Finally, the image after Gaussian filtering is separated into three color channels: red, green, and blue.

[0040] Furthermore, the modeling of the eagle-eye four-color mechanism in step three of the above scheme includes:

[0041] By processing the same signal using different cone cells in an eagle's eye, four receptor pathways were obtained: brightness pathway, long-wavelength pathway, mid-wavelength pathway, and short-wavelength pathway. Each pathway represents the contrast response of the corresponding cone cell with other types of cone cells. A perceptual model mimicking the four-color mechanism of an eagle's eye was constructed based on this contrast response.

[0042]

[0043] Where I L I M I S and I UThe output responses of the image to the four channels of long-wave, medium-wave, short-wave, and ultraviolet signals are represented, respectively. r, g, and b correspond to the red, green, and blue channels of the input image, respectively. These channels are sensitive to their respective colors. The color saliency detection results of the eagle-eye four-color mechanism are calculated.

[0044] Furthermore, the modeling of the eagle-eye four-color mechanism in step three of the above scheme also includes:

[0045] In complex environments or when the target color is impure, simply using the eagle-eye four-color mechanism is insufficient to fully capture target saliency, and further improvements are needed.

[0046] For the output I of each path k Let k = L, M, S, U. Suppose that the pixel value of one pixel p is m. Then, using Euclidean distance, we can calculate the distance D(m, n) between p and other pixels p' (corresponding to n) in terms of pixel value, and statistically analyze the histogram of the output image along its path as f. n The color saliency value I′(p) at pixel p is calculated as follows:

[0047]

[0048] By traversing every pixel in the four pathways, the color saliency detection results I′ for the four pathways are obtained. k k = L, M, S, U, and then calculate the color saliency detection result I of the eagle-eye four-color mechanism:

[0049]

[0050] Where I′ L 、I′ M 、I′ S and I′ U The results show the significance detection results for four pathways: high-wave, medium-wave, short-wave, and ultraviolet signals.

[0051] Furthermore, in the above scheme, step four, the fusion of the eagle-eye-like biconcave structure, includes...

[0052] The five-layer feature map E obtained from the network forward propagation in step one. i Let i = 1, 2, 3, 4, 5. The size of these five feature maps decreases to half that of the previous image in each subsequent layer. Among them, the higher-level feature map E5 contains global information; while the lower-level feature maps E4 and E3 contain regional information; and the lowest-level feature maps E1 and E2 contain local information. Spatial information is obtained through this relationship.

[0053] E′ i =E i *σ(Ei -E5)

[0054] Where i = 1, 2, 3, 4, σ is the sigmoid function, and * is the dot product operation. Finally, the lower-level feature map is enhanced again using the contrast information. Therefore, equation (7) obtains the spatial information E′ containing low-level local features after removing global information. i Then, the spatial information E′ i The result I is fused with the color saliency detection result I from the previous step to obtain the fused salient output I. output :

[0055] I output =E′*I

[0056] E′ is set to one or more low-level feature maps depending on the actual target situation.

[0057] Furthermore, the salience map of the aerial target using the eagle-eye vision dual-fovea analogy output in step five includes:

[0058] Through the calculations in steps one through four, the saliency detection result I for UAV identification of medium- and long-range aerial targets is obtained. output Then put I output Returning to the larger field of view based on the corresponding positional relationships, an erosion-fill operation is used to remove discontinuous small regions in the image, yielding the final saliency detection result I. output .

[0059] The advantages and effects of this invention are as follows: 1) Introducing the raptor vision mechanism into the aerial target salience detection process can better map the raptor's ability to capture mid-to-long-range prey to the task requirements of UAV mid-to-long-range target identification; 2) Convolutional neural networks require manual labeling of a large number of datasets from the pixel layer for image salience, which is a huge amount of work, while traditional target detection methods require manual setting of feature extractors. This method combines deep learning methods with traditional vision methods, obtaining the target's coordinate position in the image through training, avoiding pixel labeling while still achieving better target salience results; 3) The eagle-eye four-color mechanism has been improved by fusing salience values ​​from different channels, avoiding focusing on a single channel, and using spatial information to correct color detection results, thus improving saliency performance. Attached Figure Description

[0060] Figure 1 It is the double concave structure of raptor vision.

[0061] Figure 2 This is a flowchart of an embodiment of the algorithm of the present invention.

[0062] Figure 3 It is the feature map pyramid detection process.

[0063] Figure 4 This is the original image of Target 1.

[0064] Figure 5 This is the result image of the target detection.

[0065] Figure 6a —d represents a multi-layer feature map.

[0066] Figures 7a-d show the saliency detection results of each pathway of target one using the eagle-eye four-color mechanism.

[0067] Figures 8a-d show the saliency detection results of each pathway of target one using the improved eagle-eye four-color mechanism.

[0068] Figure 9 It is a multi-pathway fusion diagram.

[0069] Figure 10 It is a significant output image of the double-concave fusion.

[0070] Figure 11 This is the final significance result graph.

[0071] Figure 12 This is the original image for Target 2.

[0072] Figure 13 a-d shows the detection results of target two.

[0073] The labels in the diagram are explained as follows:

[0074] Deep fovea; Shallow fovea; predict; D (pixel distance); f n —Image histogram; I —Color saliency detection result; E i —Feature map of layer i; E′ i — Comparison between the feature map of the i-th layer and the global information. Detailed Implementation

[0075] The effectiveness of the proposed method will be verified below through a specific example of saliency detection of small targets at long distances.

[0076] Example 1

[0077] The experimental computer was configured with an Intel Core i7-6700 processor, 3.40GHz clock speed, 8GB of memory, and Python 3 software. The dataset used was a portion of the COCO2017 dataset containing airplanes (COCO stands for Common Objects in Context, a dataset provided by Microsoft that can be used for image recognition), consisting of 90 images.

[0078] A method for aerial target salience detection based on eagle-eye vision and dual fovea is described below. The overall technical framework of this method is as follows: Figure 3 As shown in the diagram. Research indicates that birds of prey such as eagles often use their monocular fovea to observe prey at long distances and their binocular lateral fovea to track prey at close range. Therefore, this study first simulates the fovea function of an eagle's eye, using a pyramid network structure to fuse multi-scale features to detect targets of different sizes at long distances. Once a target is detected, it is magnified, simulating the lateral fovea function of an eagle's eye. Saliency extraction is then performed on the magnified target, using a four-color mechanism similar to that of an eagle's eye. Finally, to obtain higher detection results, positional information is used to correct the color saliency map, resulting in the final saliency score.

[0079] Example 2

[0080] Based on the overall technical roadmap of Embodiment 1, the specific steps for implementing the aerial target salience detection method using a dual-fovea approach mimicking eagle-eye vision in Embodiment 2 are as follows:

[0081] Step 1: Modeling the concave mechanism of eagle eye vision

[0082] An EagleFoveaNet network, mimicking the fovea of ​​an eagle's eye, was constructed, with a 50-layer ResNet50 residual network as its backbone. Since the spatial resolution formed by the focal points of the two fovea regions in an eagle's eye is variable, a four-layer feature map pyramid network structure was built to detect targets of different sizes, as follows: Figure 4 As shown, the loss function L is defined as:

[0083] L=ΣL cls +λ1∑L reg +λ2∑L sal

[0084] Where λ1 and λ2 are constants, L cls L is the classification loss predicted by the network. reg L is the regression loss for predicting bounding boxes in the network. sal Predict the saliency loss within the bounding box for the network.

[0085] Let the significance value at target i be I, and the significance value of the currently predicted candidate box j be I′. Then the significance loss here is defined as:

[0086]

[0087] Where i and j represent the set of target pixels and the set of candidate bounding box pixels, respectively. This represents the overlapping area between the target and candidate boxes, where ||·|| represents the difference in saliency values ​​at the same location. The target is typically marked by a ground truth box, and the regression loss of the bounding box is calculated using the distance between corresponding coordinate points. However, due to the existence of symmetrical candidate boxes about the center of the ground truth box, this does not guarantee a unique candidate box. By adding a saliency loss, the saliency of symmetrical candidate boxes about the center of the ground truth box differs. If the candidate box overlaps significantly with the background of the target region, the saliency loss increases; if the candidate box overlaps significantly with the ground truth target in the target region, the saliency loss decreases. This method improves the loss function, dynamically adjusting the losses for candidate and target boxes, thereby selecting more suitable candidate boxes to detect the target.

[0088] After the input image passes through the EagleFoveaNet network, the predicted target location and category are obtained, as well as the five-layer feature map information from the forward propagation. The image within the predicted location and the predicted category are used as the input for step two.

[0089] Step 2: Modeling the foveal mechanism of eagle eye vision

[0090] To simulate the small scene of the fovea, in reality, a telephoto lens can be used to simulate the detailed features of an object observed by an eagle at close range. In the simulation, however, upsampling algorithms such as bicubic interpolation are used to simulate the fovea imaging of an eagle's eye.

[0091] Let image1 be the image within the predicted location, and image2 be the magnified image. Let F′(x,y) be the pixel value corresponding to the previous pixel in image2, where x and y are the coordinates on the image. Then, the value of F′(x,y) calculated by bitrilinear interpolation is:

[0092]

[0093] Where W(i) and W(j) are the weight coefficients of each point, and F(i,j) are the pixel values ​​of x and y mapped back to the corresponding 4×4 pixel boxes near the position in image1.

[0094] After upsampling the target region, Gaussian filtering is applied to image2. Gaussian filtering removes high-frequency components from the image, such as noise, thus smoothing the image. The image after Gaussian filtering retains low-frequency information, facilitating subsequent processing.

[0095] Finally, the image after Gaussian filtering is separated into color channels. Typically, an image consists of three color channels: r, g, and b, where r represents the red channel, g represents the green channel, and b represents the blue channel.

[0096] Step 3: Modeling based on the eagle-eye four-color mechanism

[0097] Different cone cells in an eagle's eye antagonize each other after processing the same signal, resulting in four sensory pathways: brightness pathway, long-wavelength pathway, mid-wavelength pathway, and short-wavelength pathway. Each pathway represents the contrast response of the corresponding cone cell with other types of cone cells. A perceptual model mimicking the four-color mechanism of an eagle's eye is constructed based on this contrast response.

[0098]

[0099] Where I L I M I S and I U The output responses of the image are represented by four channels: long-wave, medium-wave, short-wave, and ultraviolet signals. r, g, and b correspond to the red, green, and blue channels of the input image, respectively. Since these channels are sensitive to their respective colors and have weak responses to other color regions, they can extract the saliency of a specific color target. However, in complex environments or when the target color is impure, the eagle-eye four-color mechanism alone cannot fully capture target saliency and requires further improvement.

[0100] For the output I of each path k Let k = L, M, S, U. Suppose that the pixel value of one pixel p is m. Then, the distance D(m, n) between p and other pixels p' (corresponding to n) in terms of pixel value can be calculated using Euclidean distance. The histogram of the output image along its path is then calculated as f. n The color saliency value I′(p) at pixel p is calculated as follows:

[0101]

[0102] By traversing every pixel in the four pathways, the color saliency detection results I′ for the four pathways can be obtained. k k = L, M, S, U. Then calculate the color saliency detection result I of the eagle-eye four-color mechanism:

[0103]

[0104] Where I′ L 、I′ M 、I′ S and I′U The results show the significance of detection for four pathways: wave signal, medium wave signal, short wave signal, and ultraviolet signal.

[0105] In Equation (6), salient values ​​with large color values ​​and many pixel values ​​are highlighted, which can effectively suppress backgrounds with similar color values ​​and large areas, thus highlighting the target area. However, it only contains color information, while spatial information, such as specific features like edges and shapes, also increases salientity. Therefore, it is necessary to further integrate spatial information into the color salientity results.

[0106] Step 4: Fusion of the eagle-eye double concave shape

[0107] The five-layer feature map E obtained from the network forward propagation in step one. i Let i = 1, 2, 3, 4, 5. The size of these five feature maps decreases to half that of the previous image in each subsequent layer. Therefore, it can be determined that the higher-level feature map E5 contains global information; while the lower-level feature maps E4 and E3 contain regional information; and the lowest-level feature maps E1 and E2 contain local information. Spatial information can be obtained through this relationship:

[0108] E′ i =E i *σ(E i -E5)

[0109] Where i = 1, 2, 3, 4, σ is the sigmoid function, and * is the dot product operation. Subtracting the high-level feature map from the low-level feature map reduces global information. The sigmoid function is used to prevent negative numbers. Finally, the lower-level feature map is enhanced again using the contrast information. Therefore, equation (7) obtains the spatial information E′ containing low-level local features but without global information. i Then, the spatial information E′ i The result I is fused with the color saliency detection result I from the previous step to obtain the fused salient output I. output :

[0110] I output =E′*I

[0111] E′ is set to one or more low-level feature maps depending on the actual target situation.

[0112] Step 5: Output a salience map of aerial targets using a dual-fovea model similar to eagle-eye vision.

[0113] Through the calculations in steps one through four, the saliency detection result I for UAV identification of medium- and long-range aerial targets is obtained. output Then put I outputReturning to the larger field of view based on the corresponding positional relationships, an erosion-fill operation is used to remove discontinuous small regions in the image, yielding the final saliency detection result I. output Based on the target detection results, the drone is further guided to fly towards the target and perform subsequent tasks.

[0114] Example 3

[0115] The specific steps of a method for detecting aerial target salience using a double fovea-like vision, mimicking eagle eye vision, are as follows:

[0116] Step 1: Modeling the concave mechanism of eagle eye vision

[0117] The EagleFoveaNet network is initialized, with pre-trained model parameters loaded onto the backbone. These pre-trained parameters are publicly available and pre-trained on the COCO-val2017 dataset provided by PyTorch. The input image format is image(b,h,w,c), where b represents the batch size, h represents the height, w represents the width, and c represents the number of channels. For each pixel in the image, 15 anchor boxes are generated according to the following size and proportions:

[0118] anchor_size=(32,64,128,256,512)

[0119] aspect_ratios = (0.5, 1.0, 2.0)

[0120] The EagleFoveaNet network was trained. The dataset was divided into training and test sets in a 4:1 ratio. The training set was input into the network, and candidate boxes were generated at each pixel according to equations (9)-(10). The Intersection over Union (IoU) ratio was used to measure the difference between the candidate box and the ground truth box. IoU is the ratio of the area of ​​overlap between the candidate box and the ground truth box to the total area of ​​the two. If the IoU between the candidate box and the ground truth box is greater than 0.75, the candidate box is considered a positive sample; if the IoU is less than 0.3, the candidate box is considered a negative sample, and those in between are not used. After screening the candidate boxes in the above way, the loss function was calculated for each prediction result according to equation (1). Then, the loss value was backpropagated, and the parameters in the network were adjusted by stochastic gradient descent to train the network. The training step size was 100 steps, and the parameters of stochastic gradient descent were set as follows: learning rate 0.0025, momentum 0.9, and weight decay 0.0005. After training is complete, save the model weight coefficients as resnet50_fpn.pth. In subsequent operations, load these weight parameters directly.

[0121] The test set uses instance images of size image(1,372,640,3), such as... Figure 4As shown. First, load the model's weight coefficients resnet50_fpn.pth, and input the image into... network The forward propagation is performed. The results of the forward propagation are output as a list, where each element is a dictionary containing `boxed` (the predicted bounding box), `labels` (the category of the bounding box), and `scores` (confidence scores). In this example, `boxes` is an array (14, 4), indicating that the network detected 14 possible target regions, with the four columns representing the xmin, ymin, xmax, and ymax of the x and y axes, respectively. `labels` is an array (14,) with the same number of rows as `boxes`, indicating the target category for each possible target region; since only one target is included here, all rows are set to 1. `scores` has the same number of rows as `labels`, representing the confidence score for each predicted category. Bounding boxes with a confidence score greater than 0.8 are selected as target regions and bounded according to the coordinates of `boxes`, as shown below. Figure 5 As shown, the results demonstrate that the network can effectively distinguish between aircraft at different distances, while the confidence score for a bird on the left side of the image is 0.7699, allowing it to be easily removed using the threshold. This example highlights the network's advantages, including high detection accuracy and the ability to detect targets at different scales.

[0122] Simultaneously, feature maps for each layer are obtained and stored as an ordered dictionary, representing feature map information from low to high levels. In this example, because the backbone network will reduce the image size to its original value... Therefore, the size of the first layer feature map is (1, 256, 93, 160). It's important to note that the output of the backbone is in the form of (b, c, h, w), which is slightly different from the input format (b, h, w, c). Then, the width and height of each layer feature map are respectively those of the previous layer. The last four layers are (1,256,47,80), (1,256,24,40), (1,256,12,20), and (1,256,6,10). Since the feature maps have more channels than the original image, they need to be added along the channel dimension. Furthermore, because the feature map sizes are inconsistent across layers, the higher-level feature maps need to be scaled down to the lower-level feature maps. The result for each layer is as follows: Figure 6a As shown in Figure 6, from left to right, the feature maps represent feature maps from high to low levels. A comparison reveals that the closer to the low level, the more clearly the specific location of the detected target is highlighted. For example, in the rightmost feature map in Figure 6, the larger the feature value of a pixel (i.e., the brighter the color), the more likely it is to belong to the target. This is consistent with... Figure 5 The detection results were consistent, thus confirming that the feature map contains spatial information about the target.

[0123] Step 2: Modeling the foveal mechanism of eagle eye vision

[0124] The target region obtained in step one needs to be upsampled using bicubic interpolation to simulate foveal imaging in eagle-eye vision. Since the target region image does not have the same aspect ratio as the original image, multiple upsampling will distort the image. Therefore, only one cubic linear interpolation is used to simulate foveal imaging at close range in eagle-eye vision. For the example... Figure 5 The detection results are used to calculate the color saliency of the detection box with the highest confidence. The detection box size is (1, 118, 148, 3), where 1 represents the batch size, 118 represents the image height, 148 represents the image width, and 3 represents the number of image channels. Then, a Gaussian filter is applied to this image region, with a mean of 0 and a variance of 1.5. Finally, the image after Gaussian filtering is separated into three color channels: red (r), green (g), and blue (b).

[0125] Step 3: Modeling based on the eagle-eye four-color mechanism

[0126] Let I be the color pathways of the target region image obtained before the calculation by the eagle-eye four-color mechanism perception model. L I M I S and I U For example Figure 5 The detection results are used to calculate the color saliency of the detection box with the highest confidence. At this time, the size of the detection box is (1, 118, 148, 3). First, the r, g, and b channels of the image color separation are calculated according to Equation (4). L I M I S and I U Each pathway has a size of (118, 148). The results for the four pathways are as follows. Figure 7a As shown in Figure d, from left to right, the output responses of the four signal channels—long-wave, medium-wave, short-wave, and ultraviolet—to the image are respectively displayed. It can be observed that the target is mostly concentrated in the short-wave and ultraviolet signals, while the background is concentrated in the long-wave signal. However, if we change the image example, such as... Figure 12 In the example shown, the signals of each channel are as follows: Figure 13a As shown in —d, the target is concentrated in the mid-wave and ultraviolet signals. Through this comparison, it was found that using only the four-color mechanism of the eagle eye simulation cannot fully obtain the target salience. When the target changes, the target's color space needs to be determined in advance, and this prior information cannot be obtained beforehand.

[0127] Therefore, it is also necessary to improve the color of each channel and calculate the histogram f of the feature map of each channel. n And the color distance D(m,n) of each point from other points, for this instance, f nBoth D(m,n) and D(m,n) are of the form (256,). D(m,n) calculates the color distance between color values ​​m and n, where n = 0-255. The saliency map is calculated according to equation (5), with a size of (118,148,4). Figure 8a As shown in Figure d, from left to right, the output responses of the four pathways—improved long-wavelength signal, improved mid-wavelength signal, improved short-wavelength signal, and improved ultraviolet signal—to the image are represented. Comparison with Figure 7 reveals that the color contrast effectively suppresses large areas of background color in these pathways.

[0128] Finally, according to formula (6), the color saliency of each channel is merged, such as... Figure 9 As shown, this step can suppress the background color and preserve the target salience value.

[0129] Step 4: Fusion of the eagle-eye double concave shape

[0130] According to Figure 5 The selected target region (1,118,148,3) is used to obtain the feature map of the target region from the coordinate position of the target region in Figure 6. The background position color feature value is corrected using spatial information to enhance the color feature value of the target position. In this example, the size of feature map E5 is (1,1,6,10), and the size after one upsampling is (1,1,12,20). Feature map E4 is taken, and the size of the comparison feature map E′4 is calculated according to formula (7), which is still (1,1,12,20). At this time, the comparison feature map is subjected to bicubic interpolation upsampling operation again to make the size return to (1,1,372,640) to obtain the region information. The above operation is also applied to feature maps E3, E2 and E1 to obtain the spatial information of each level of the whole image. It should be noted that feature map E5 needs to be subjected to four bicubic interpolation upsampling operations to ensure that the image size is the same when calculated according to formula (7). In this example, E′ selects all low-level spatial information, so for different E′ i For i = 1, 2, 3, 4, it is necessary to upsample to the original image size and then add them together to obtain E′. Finally, calculate the final significant output I according to equation (8). output .

[0131] In the example, E′ i The elements in i = 1, 2, 3, 4 may contain negative numbers. Therefore, the calculation result of equation (8) needs to be made so that all numbers less than 0 are 0, to ensure that the value of the image falls within the correct range and to prevent overflow.

[0132] Finally, the information from the target region on the feature map is reused to perform binarization segmentation on the image. The goal is to apply a higher threshold to salient regions of the target area and a lower threshold to non-salient regions. Therefore, in this example, the regions in E′ where the target may appear are traversed, and the lowest feature value is found to be 0.1. Thus, binarization is performed with a value of 0.1, as shown below. Figure 10 As shown.

[0133] Step 5: Output a salience map of aerial targets using a dual-fovea model similar to eagle-eye vision.

[0134] Through the calculations in steps one through four, the saliency detection result I for UAV identification of medium- and long-range aerial targets is obtained. output Then put I output Returning to the larger field of view based on the corresponding positional relationships, a 3×3 element was selected as the structuring element for erosion-fill morphological operations. This involved filling once, eroding once, and then filling again. This process removed small protrusions and filled small holes. Finally, the salience map was returned to the concave center of the simulated eagle-eye vision, as shown in the image. Figure 11 As shown, the results demonstrate that the improved eagle-eye four-color mechanism and feature map spatial information can fully display the salient map of the entire aircraft.

[0135] The above embodiments further illustrate that the aerial target salience detection method based on the eagle eye vision dual fovea of ​​the present invention simulates the dual fovea mechanism of the eagle eye, improves the loss function, uses a feature map pyramid network structure to detect targets at different scales, enriches the detailed features of the detection results, uses the eagle eye four-color mechanism to calculate color salience values ​​for different wavelength signals, and finally uses spatial information to correct the color detection results to achieve salience detection of distant aerial targets.

Claims

1. A method for detecting the salience of aerial targets using a bifocal approach mimicking eagle-eye vision, characterized by: The steps include the following: Step 1: Modeling the concave mechanism of eagle eye vision A network model of the central concave shape of eagle eye vision is established. The backbone of the network is a 50-layer residual network. After the input image passes through the network, the predicted target location and type are obtained, as well as the five-layer feature map information of forward propagation. The image within the predicted location and the predicted type are used as the input for step two. Step 2: Modeling the foveal mechanism of eagle eye vision After upsampling the target region, the image is processed by Gaussian filtering to retain low-frequency information. Finally, the color channels of the Gaussian-filtered image are separated. Step 3: Modeling based on the eagle-eye four-color mechanism By analyzing the mutual antagonism of different cone cells in the eagle eye after processing the same signal, four receptor pathways were obtained. Each pathway represents the contrast response of the corresponding cone cell with other types of cone cells. A perception model of the eagle eye four-color mechanism was constructed based on this contrast response. The distance between the signal pixel values ​​of each pathway was calculated, and the histogram of the output image of each pathway was statistically analyzed. The color saliency detection results of the eagle eye four-color mechanism were calculated. Step 4: Fusion of the eagle-eye double concave shape Upsampling is performed using the multi-layer feature map information obtained from the network forward propagation in step one to calculate spatial information. Then, the spatial information is fused with the color saliency detection result from the previous step to obtain a fused saliency output. Step 5: Output a salience map of aerial targets using a dual-fovea model similar to eagle-eye vision. The obtained saliency outputs are returned to the large field of view according to their corresponding positional relationships. The erosion-fill operation is used to remove discontinuous small regions in the image to obtain the final saliency detection result. The modeling of the foveal mechanism mimicking the eagle eye's visual side in step two includes: By using a telephoto lens to simulate the detailed features of objects observed by an eagle at close range, that is, to simulate the small scene of the fovea on the visual side of the eagle's eye, and to simulate the fovea imaging on the visual side of the eagle's eye by using a bicubic interpolation upsampling algorithm. Let image1 be the image within the predicted location, and image2 be the magnified image. Let F′(x,y) be the pixel value corresponding to the previous pixel in image2, where x and y are the coordinates on the image. Then, the value of F′(x,y) calculated by bitrilinear interpolation is: Where W(i) and W(j) are the weight coefficients of each point, and F(i,j) are the pixel values ​​of x and y mapped back to the corresponding 4×4 pixel boxes near the position in image1; After upsampling the target region, Gaussian filtering is performed on image2 to remove high-frequency components and retain low-frequency information. Finally, the image after Gaussian filtering is separated into three color channels: red, green, and blue.

2. The aerial target salience detection method based on eagle-eye vision and double fovea as described in claim 1, characterized in that: The modeling of the concave mechanism of eagle eye vision in step one includes: An EagleFoveaNet network, mimicking the fovea of ​​an eagle's eye, was constructed. The backbone of the network was a 50-layer ResNet50 residual network. Based on the variable spatial resolution formed by the focal points of the two fovea regions in an eagle's eye, a four-layer feature map pyramid network structure was established to detect targets of different sizes. The loss function L was defined as: L=∑L cls +λ1∑L reg +λ2∑L sal Where λ1 and λ2 are constants, L cls L is the classification loss predicted by the network. reg L is the regression loss for predicting bounding boxes in the network. sal Predict the salient loss within the bounding box for the network; Let the significance value at target i be I, and the significance value of the currently predicted candidate box j be I′. Then the significance loss here is defined as: Where i and j represent the set of target pixels and the set of candidate bounding box pixels, respectively. This represents the overlapping area between the target and the candidate box, where ||·|| is the difference in significance values ​​at the same location. After the input image passes through the EagleFoveaNet network, the predicted target location and category are obtained, along with the five-layer feature map information from the forward propagation. The image within the predicted location and the predicted category are then used as input for step two.

3. The aerial target salience detection method based on eagle-eye vision and double fovea as described in claim 1, characterized in that: The modeling of the eagle-eye four-color mechanism in step three includes: By processing the same signal using different cone cells in an eagle's eye, four receptor pathways were obtained: brightness pathway, long-wavelength pathway, mid-wavelength pathway, and short-wavelength pathway. Each pathway represents the contrast response of the corresponding cone cell with other types of cone cells. A perceptual model mimicking the four-color mechanism of an eagle's eye was constructed based on this contrast response. Where I L I M I S and I U The output responses of the image to the four channels of long-wave, medium-wave, short-wave, and ultraviolet signals are represented, respectively. r, g, and b correspond to the red, green, and blue channels of the input image, respectively. These channels are sensitive to their respective colors. The color saliency detection results of the eagle-eye four-color mechanism are calculated.

4. The aerial target salience detection method based on eagle-eye vision and double fovea as described in claim 3, characterized in that: The modeling of the eagle-eye four-color mechanism in step three also includes: For the output I of each path k Let k = L, M, S, U. Let the pixel value of one pixel p be m. Then, calculate the distance D(m, n) from other pixels p' in terms of pixel value using Euclidean distance, and statistically analyze the histogram of the output image of the path as f. n The color saliency value I′(p) at pixel p is calculated as follows: By traversing every pixel in the four pathways, the color saliency detection results I for the four pathways are obtained. k ′, k=L,M,S,U, and then calculate the color saliency detection result I of the eagle-eye four-color mechanism: Where I′ L 、I′ M 、I′ S and I′ U The results show the significance detection results for four pathways: long-wave, medium-wave, short-wave, and ultraviolet signals.

5. The aerial target salience detection method based on eagle-eye vision and double fovea as described in claim 1, characterized in that: Step four, the fusion of the eagle-eye-like double concave shape, includes: The five-layer feature map E obtained from the network forward propagation in step one. i Let i = 1, 2, 3, 4, 5. The size of these five feature maps decreases to half that of the previous image in each subsequent layer. Among them, the higher-level feature map E5 contains global information; while the lower-level feature maps E4 and E3 contain regional information; and the lowest-level feature maps E1 and E2 contain local information. Spatial information is obtained through this relationship. E i ′=E i *σ(E i -E5) Where i = 1, 2, 3, 4, σ is the sigmoid function, * is the dot product operation, and finally, the lower-level feature maps are enhanced using contrast information to obtain spatial information E containing low-level local features but without global information. i Then, the spatial information E i The result is fused with the color saliency detection result I from the previous step to obtain the fused salient output I. output : I output =E′*I E′ is set to one or more low-level feature maps depending on the actual target situation.

6. The aerial target salience detection method based on eagle-eye vision and double fovea as described in claim 5, characterized in that: The output of step five, a salience map of aerial targets with eagle-eye vision and double fovea, includes: Through the calculations in steps one through four, the saliency detection result I for UAV identification of medium- and long-range aerial targets is obtained. output Then put I output Returning to the larger field of view based on the corresponding positional relationships, an erosion-fill operation is used to remove discontinuous small regions in the image, yielding the final saliency detection result I. output .

Images