Remote sensing image target detection method and system

Abstract

The present application discloses a method and system for remote sensing image target detection. The method comprises: generating target candidate frames using a dual-channel vision mechanism candidate frame selection model for remote sensing images, cropping the remote sensing images according to the target candidate frames; inputting the cropped images into a trained target detection model for target recognition, and outputting the target's bounding box and probability. This system can select target candidate frames for remote sensing images of different scales, efficiently predicting the target's position and size, and improving the intelligent detection capabilities of multi-scale remote sensing images.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of remote sensing image processing, and specifically to a remote sensing image target detection method and system. Background Art

[0002] With the rapid development of remote sensing technology, the quantity and quality of remote sensing image data have exploded. Faced with this massive amount of remote sensing data, traditional visual interpretation methods are no longer able to accurately and real-timely extract geographic information. Therefore, intelligent remote sensing interpretation models are needed to rapidly and accurately interpret remote sensing imagery. Remote sensing image target detection is a crucial step in extracting information from remote sensing imagery, enabling image interpretation and target monitoring. It directly impacts the intelligent interpretation capabilities of remote sensing images and the utilization rate of remote sensing data.

[0003] Although researchers have designed powerful feature extraction and representation models to address the unique challenges of remote sensing target detection, some problems remain unsolved. Targets appear in different sizes and poses across remote sensing images, making feature extraction and transfer difficult. This is the primary reason for the difficulty in improving remote sensing target recognition accuracy and significantly impacts the algorithm's transferability. Summary of the Invention

[0004] To this end, the embodiments of the present application provide a remote sensing image target detection method and system, which aims to address the problem of scale changes in remote sensing images and solve the problems of fixed size and fixed step length in traditional candidate frame selection. It can select target candidate frames on remote sensing images of different scales, efficiently complete the prediction of the target's position and size, and improve the intelligent detection capability of multi-scale remote sensing image targets.

[0005] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a remote sensing image target detection method is provided, the method comprising:

[0007] A dual-channel vision mechanism candidate frame selection model is used to generate a target candidate frame for the remote sensing image, and the remote sensing image is cropped according to the target candidate frame;

[0008] The cropped image is input into the trained object detection model for object recognition, and the bounding box and probability of the object are output.

[0009] Optionally, the dual-channel visual mechanism candidate box selection model includes a visual saliency calculation model and an attention saliency calculation model;

[0010] The method of using a dual-channel vision mechanism candidate frame selection model to generate a target candidate frame for a remote sensing image includes:

[0011] Determining the position of a target candidate frame for the remote sensing image based on the attention saliency calculation model;

[0012] Determining the size of an object in the remote sensing image based on the visual saliency calculation model;

[0013] The target candidate box is adaptively generated according to the position of the target candidate box and the size of the target.

[0014] Optionally, determining the size of the target in the remote sensing image based on the visual saliency calculation model includes:

[0015] Converting the remote sensing image into a visual saliency map;

[0016] Use grayscale segmentation threshold to convert the salient areas in the visual salient map into visually salient objects;

[0017] Use the local maximum method to extract fixation points from the attention saliency map;

[0018] A visually salient object that coincides with the gaze point is selected to generate a bounding rectangular frame of each visually salient object, and the size of the target is determined according to the bounding rectangular frame.

[0019] Optionally, determining the position of the target candidate frame based on the attention saliency calculation model for the remote sensing image includes:

[0020] Converting the remote sensing image into an attention saliency map;

[0021] Use the local maximum method to extract fixation points from the attention saliency map;

[0022] Use grayscale segmentation threshold to convert the salient areas in the visual salient map into visually salient objects;

[0023] All gaze points are regarded as target positions. If the gaze point is within a visually salient object, the position of the target candidate box is determined based on the visually salient object; if the gaze point is not within the visually salient object, the position of the target candidate box is determined with the gaze point as the center.

[0024] Optionally, the attention saliency calculation model is constructed according to the following steps:

[0025] Pre-training an attention saliency calculation model using a remote sensing image eye movement dataset, wherein the remote sensing image eye movement dataset includes geospatial images of maps, remote sensing images, and street scenes; wherein the attention saliency calculation model is loaded with pre-trained weights of a VGG-16 convolutional neural network, and a data augmentation method is used to expand the size of the remote sensing image eye movement dataset;

[0026] The eye movement data collected in the experiment and the corresponding remote sensing images are used to train and test the attention saliency calculation model.

[0027] Optionally, the target detection model is constructed according to the following steps:

[0028] Select remote sensing images of various scales as stimulus materials to obtain eye movement and brain neural data when users perform visual target detection;

[0029] Analyze the user's eye movement patterns and neural responses when identifying the target object based on eye movement and brain neural data, extract the target objects that are easy to make mistakes and difficult to distinguish, and construct a difficult-to-distinguish sample set;

[0030] The Faster-RCNN model is trained based on the difficult-to-classify sample set to obtain the target detection model.

[0031] Optionally, the method further includes:

[0032] Calculate recall and precision based on the bounding box and probability of the target;

[0033] The object detection model is optimized based on recall and precision.

[0034] According to a second aspect of an embodiment of the present application, a remote sensing image target detection system is provided, the system comprising:

[0035] A target candidate frame module is used to generate a target candidate frame for the remote sensing image using a dual-channel visual mechanism candidate frame selection model, and to crop the remote sensing image according to the target candidate frame;

[0036] The target recognition module is used to input the cropped image into the trained target detection model for target recognition and output the target's bounding box and probability.

[0037] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0038] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.

[0039] In summary, the embodiments of the present application provide a method and system for remote sensing image target detection. This system uses a dual-channel vision mechanism candidate frame selection model to generate target candidate frames for remote sensing images, then crops the remote sensing images according to the target candidate frames. The cropped images are then input into a trained target detection model for target recognition, outputting the target's bounding box and probability. This system is capable of selecting target candidate frames for remote sensing images of varying scales, efficiently predicting the target's position and size, and enhancing the intelligent detection capabilities of multi-scale remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0041] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0042] Figure 1 A flowchart of a remote sensing image target detection method provided in an embodiment of the present application;

[0043] Figure 2 Schematic diagram of the overall research framework provided for the embodiments of this application;

[0044] Figure 3 A schematic diagram of a candidate box selection model based on a dual-channel vision mechanism provided in an embodiment of the present application;

[0045] Figure 4 A block diagram of a remote sensing image target detection system provided in an embodiment of the present application;

[0046] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown;

[0047] Figure 6 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0049] Figure 1 The remote sensing image target detection method provided in an embodiment of the present application is shown, and the method includes the following steps:

[0050] Step 101: generating a target candidate frame for the remote sensing image using a dual-channel vision mechanism candidate frame selection model, and cropping the remote sensing image according to the target candidate frame;

[0051] Step 102: Input the cropped image into the trained object detection model for object recognition, and output the bounding box and probability of the object.

[0052] In a possible implementation, the dual-channel visual mechanism candidate box selection model includes a visual saliency calculation model and an attention saliency calculation model.

[0053] In step 101, the method of generating a target candidate frame using a dual-channel vision mechanism candidate frame selection model for a remote sensing image includes:

[0054] The position of the target candidate frame is determined for the remote sensing image based on the attention saliency calculation model; the size of the target is determined for the remote sensing image based on the visual saliency calculation model; and the target candidate frame is adaptively generated according to the position of the target candidate frame and the size of the target.

[0055] In a possible implementation, determining the size of the target in the remote sensing image based on the visual saliency calculation model includes:

[0056] The remote sensing image is converted into a visual saliency map; a salient area in the visual saliency map is converted into a visual salient object using a grayscale segmentation threshold; a fixation point is extracted from the attention saliency map using a local maximum method; a visual salient object that coincides with the fixation point is selected to generate a bounding rectangle for each visual salient object, and the size of the target is determined based on the bounding rectangle.

[0057] In a possible implementation, determining the position of the target candidate frame for the remote sensing image based on the attention saliency calculation model includes:

[0058] The remote sensing image is converted into an attention saliency map; a local maximum method is used to extract fixation points from the attention saliency map; a grayscale segmentation threshold is used to convert a salient area in the visual saliency map into a visually salient object; all fixation points are regarded as target positions, and if the fixation point is within the visually salient object, the position of the target candidate box is determined based on the visually salient object; if the fixation point is not within the visually salient object, the position of the target candidate box is determined with the fixation point as the center.

[0059] In one possible implementation, the attention saliency calculation model is constructed according to the following steps:

[0060] An attention saliency calculation model is pre-trained using a remote sensing image eye movement dataset; the remote sensing image eye movement dataset includes geospatial images of maps, remote sensing images, and street scenes. The attention saliency calculation model is loaded with pre-trained weights of a VGG-16 convolutional neural network, and the number of remote sensing image eye movement datasets is expanded using a data augmentation method. The attention saliency calculation model is trained and tested using the experimentally collected eye movement data and the corresponding remote sensing images.

[0061] In one possible implementation, the target detection model in step 102 is constructed according to the following steps:

[0062] Remote sensing images of various scales are selected as stimulus materials to obtain the user's eye movement and brain nerve data when performing visual target detection. Based on the eye movement and brain nerve data, the user's eye movement patterns and neural responses when identifying targets are analyzed, and targets that are easy to misidentify and difficult to distinguish are extracted to construct a difficult-to-distinguish sample set. The Faster-RCNN model is trained based on the difficult-to-distinguish sample set to obtain a target detection model.

[0063] In a possible implementation, after step 102, the method further includes:

[0064] Calculating recall and precision based on the bounding box and probability of the target; and optimizing the target detection model based on the recall and precision.

[0065] The embodiments of the present application aim to solve the problem that remote sensing images have large scale variations and traditional target detection models have difficulty in effectively identifying targets. By designing and implementing eye movement-EEG target detection cognitive experiments, the multi-scale target retrieval cognitive laws of professional interpreters are explored, and the target search cognitive mechanism of researchers is studied. Based on the target search cognitive laws, a candidate box selection model based on a dual-channel visual attention mechanism is proposed, which realizes the fast and accurate selection of candidate boxes. By jointly analyzing eye movement and EEG data, the target recognition cognitive mechanism of researchers is studied, and a difficult-to-separate sample data set based on the target recognition mechanism is constructed, which improves the recognition accuracy of the detection model and realizes target detection in multi-scale remote sensing images. The cognitive mechanism of target search and recognition of researchers is revealed, and a target detection model is constructed based on the cognitive mechanism, which realizes target detection in multi-scale remote sensing images. The results of the embodiments of the present application prove that the remote sensing image target detection method based on human intelligence has good detection accuracy and generalizability.

[0066] In response to the problem that remote sensing images have large scale variations and traditional target detection models have difficulty in effectively identifying targets, starting from the human visual cognitive mechanism, by designing and implementing eye movement-EEG target detection cognitive experiments, we explore the multi-scale target retrieval cognitive laws of professional interpreters and study the target search cognitive mechanism of researchers; based on the target search cognitive laws, we propose a candidate box selection model based on a dual-channel visual attention mechanism, which realizes the fast and accurate selection of candidate boxes; by jointly analyzing eye movement and EEG data, we study the target recognition cognitive mechanism of researchers, construct a difficult-to-separate sample data set based on the target recognition mechanism, improve the recognition accuracy of the detection model, and realize target detection in multi-scale remote sensing images.

[0067] Figure 2 The research framework for target recognition provided by the embodiment of the present application is shown. First, for remote sensing images of different scenes and scales, the cognitive process of professionals when interpreting them is explored, and cognitive laws are studied. In this part, mainly by performing eye tracking and electroencephalogram (EEG), selecting remote sensing images of various scales as stimulus materials, using aircraft as search targets, recruiting personnel with rich visual interpretation experience and relevant professional backgrounds as researchers, carrying out experimental tasks of target search and recognition, obtaining the researchers' cognitive data when performing visual target detection, and analyzing the eye movement and brain nerve data to study the researchers' target detection cognitive process and summarize the researchers' cognitive laws.

[0068] To address the issues of a large number of candidate boxes, slow selection speed, and difficulty adapting to multi-scale targets, this method leverages the human target search attention mechanism to select the most relevant eye movement features in the image to represent human attention results. This eye movement feature is then used with remote sensing imagery to train a deep learning model, achieving "top-down" target attention saliency prediction. Simultaneously, a computational model is used to calculate the visual saliency of the image, simulating "bottom-up" visual saliency prediction. Leveraging the human dual-channel attention mechanism, this method integrates attentional saliency and visual saliency to quickly and accurately select candidate boxes.

[0069] To address the problem of some objects being easily misidentified in remote sensing target detection, by mining eye movement data and EEG data, analyzing the eye movement patterns and neural responses of professionals when identifying objects, extracting targets that are easy to misidentify, constructing a difficult-to-distinguish sample set, and through intensive training of the target detection model, reducing the model's recognition error rate and achieving accurate target identification.

[0070] For remote sensing images of different scales, researchers are influenced by both the task target features and the image features, which means that their target detection is regulated by both top-down and bottom-up visual attention mechanisms. Since the position changes of adjacent fixations are achieved by saccades, and the number of fixations is strongly correlated with target features, the number and position of fixations can simultaneously represent the researchers' scanning and fixation behaviors on the target. Color is the primary feature used by researchers to search and identify targets, followed by edge features.

[0071] For the task of target detection in remote sensing images, the visual attention guidance mechanism of researchers is studied. By comparing the visual saliency of remote sensing images, the semantic saliency of targets and the attention saliency of researchers, a theoretical basis is provided for the candidate box selection model. Based on the visual guidance mechanism, computational models and deep learning methods are used to simulate human visual search, and a human gaze point prediction model for remote sensing images is constructed to provide a data basis for the candidate box selection model. The human gaze point prediction model is used to predict the areas where targets in remote sensing images may exist, a candidate box selection model is constructed, and the target verification data in the dataset is used for testing.

[0072] Real life contains a vast amount of visual information, but human visual perception and cognition are limited in their processing capacity. Humans control eye movements to focus, acquire visual features in scene regions containing relevant information, identify their semantic information, and encode it into short-term and long-term memory. They select the information most relevant to the current task for processing and understanding. The same is true for remote sensing imagery. Human visual attention is regulated simultaneously by two pathways: bottom-up and top-down. When viewing images freely, humans are drawn to certain "special" elements within the image. These elements are distinguished by their intrinsic properties—color, shape, and texture—relative to the background. The locations of these elements are called visually salient areas in the image. This purely externally driven attentional guidance is bottom-up visual attention. When humans need to complete a task, they selectively focus on task-relevant elements within the image. This selective attention is top-down visual attention, a form of internal attentional guidance based on prior knowledge, pre-planning, and current goals.

[0073] A saliency map is an image that highlights the areas that our eyes focus on first. It typically uses features like brightness, edges, and color to determine the difference between the target area and its surroundings, and then calculates its saliency. This embodiment uses the Itti model to calculate image saliency.

[0074] An attention saliency map is a method used to describe the distribution of fixations in eye movement experiment data. It can show the density distribution of the areas that multiple subjects are looking at on the screen. The fixation kernel density map can be used to evaluate the visual attention of subjects and the salient areas in visual tasks. In the fixation kernel density map, each fixation point is regarded as the center of a Gaussian distribution, and the variance of the center represents the size of the fixation point. The calculation method of the fixation kernel density map is to superimpose the Gaussian distribution functions of all fixations to obtain a density function, and then use a kernel function (usually a normal distribution) to smooth the density function to better display the density distribution.

[0075] Neither attention saliency nor visual saliency can independently predict the complete object semantic saliency, but when added together they can predict all object locations.

[0076] The construction of the candidate box selection model of the dual-channel visual attention mechanism provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0077] Figure 3This paper presents a candidate box selection model that integrates bottom-up and top-down visual attention mechanisms. This model uses the attention saliency prediction results of MSI-Net and the visual saliency calculation process of Itti as input. On the one hand, it uses the powerful position prediction ability of the attention saliency map to determine the position of the candidate box. On the other hand, it extracts the size of the target through visual saliency, thus constructing a candidate box selection model based on a dual-channel visual mechanism. In the first step, the model uses the Itti model and the MSI-Net model to calculate the visual saliency map and the attention saliency map, respectively. In the second step, the visual saliency and attention saliency are integrated.

[0078] The saliency-based method is based on visual saliency. It uses grayscale threshold segmentation to convert the salient areas in the visual saliency map into visually salient objects. At the same time, it uses the local maximum method to extract the fixation points from the attention saliency map and selects the objects that coincide with the fixation points to generate an circumscribed rectangular box, which is the candidate box.

[0079] Attention-based methods focus on attention saliency, using a local maximum method to extract fixations from the attention saliency map. They also use grayscale threshold segmentation to convert salient regions in the visual saliency map into visually salient objects. All fixations are considered target locations. If a fixation point is within a visually salient object, a candidate bounding box is generated using that object. If a fixation point is not within a visually salient object, a candidate bounding box is generated centered on the fixation point.

[0080] The key to candidate box selection is the position and size of the target. The embodiment of the present application studies the relationship between the attention saliency map and the visual saliency map and the size and position of the target.

[0081] First, the local maximum method is used to extract fixation points from the visual saliency map and the attention saliency map. A local maximum refers to a region where the value of a point in the region is greater than the values ​​of all other points in the region. The specific steps are:

[0082] ① Define the minimum distance, which is the calculation range of the local maximum value;

[0083] ② Traverse all pixels. If the pixel value is the only maximum value within the minimum distance, then the point is a local maximum.

[0084] ③ Record the locations of all local maxima, which are all the local maxima in the heat map.

[0085] The total number of fixations refers to the sum of all fixations extracted from the saliency map. The number of intra-target fixations refers to the sum of the fixations extracted from the saliency map that fall within the target range. If a fixation point exists within the target range, the target location is considered "fixated." The total number of "fixated" targets in each image is the number of fixations. The fixation recall is the ratio of the number of fixations to the total number of targets in the image, indicating the accuracy of the saliency map in predicting target locations. The fixation precision is the ratio of the number of intra-target fixations to the total number of fixations, indicating the efficiency of the saliency map in predicting target locations.

[0086] Subsequently, the attention saliency map and the visual saliency map are segmented into rectangular boxes using a grayscale threshold segmentation method to identify areas where the target may be present. In the embodiment of the present application, a triangulation method is used to automatically select the segmentation threshold, which uses a geometric method based on the histogram to calculate the optimal segmentation threshold.

[0087] In order to statistically analyze the prediction effect of the saliency map on the target size, that is, the degree of proximity between the rectangular box generated by the saliency map and the size of the target verification box. After the grayscale threshold segmentation is completed, the binary image is converted into a rectangular box, and the IoU (Intersection over Union) between the rectangular box and the target verification box is calculated. The rectangular box with an IoU greater than 0.5 is regarded as the pre-selected box of the target, and the average area of ​​the pre-selected box on each image is counted. Finally, the difference between the average area of ​​the pre-selected box and the average area of ​​the target is calculated. In order to represent the difference between the average area of ​​the pre-selected box and the average area of ​​the target, the embodiment of the present application designs a target area similarity index, and the calculation formula is as follows:

[0088]

[0089] Among them, A t is the average area of ​​the target in the image, A c is the average area of ​​the candidate boxes. A greater similarity indicates a closer match between the candidate box and the target size. The results show that the target area similarity of the visual saliency map is higher (0.401), while the similarity of the attention saliency map is lower (0.334). The results indicate that the visual saliency map is more effective in predicting target size. This may be because visual saliency can detect edge information in remote sensing imagery and identify target boundaries. However, since human attention is not focused on a single part of the target, especially when the target is large, the attention saliency is more concentrated, resulting in a weaker prediction of target size.

[0090] In summary, attentional saliency prediction can effectively predict the location of the target. However, due to the human attention mechanism, the focus tends to be localized on the target, so it cannot accurately predict the target's size. Visual saliency prediction, on the other hand, detects all objects with high contrast against the background. The highly salient areas in its calculation result contain both the target and other salient objects, so the similarity with the target distribution is low. However, thanks to its good edge feature calculation, the Itti model can well represent the size and edges of different objects.

[0091] When using Itti to calculate an image's visual saliency map, setting the input image size parameter can control the map's resolution and computation speed. When extracting fixations from the attention saliency map using the local maximum method, the location and number of fixations can be controlled by setting a minimum distance. This section discusses the impact of different methods and parameters on the accuracy of object position prediction. Prediction results are evaluated using Intersection over Union (IoU).

[0092] IoU (Intersection over Union), also known as intersection over union, is a measure of the accuracy of detecting corresponding objects in a specific dataset. It is the result of dividing the overlapping part of two regions by the combined part of the two regions. The larger the IoU, the closer the prediction result is to the true value. The calculation formula is:

[0093]

[0094] Among them, A overlap Represents the intersection area of ​​two boxes, A union Represents the area of ​​the union of two boxes.

[0095] The visual saliency selection method is a method that only uses the visual saliency map to generate candidate boxes. First, the histogram is used to Figure 3 The angle method determines the segmentation threshold, which is then used to extract salient objects from the visual saliency map. Finally, a bounding rectangle is generated for each salient object. The attention saliency selection method uses only the attention saliency map to generate candidate boxes. First, the local maximum method is used to extract fixation points from the attention saliency map, and then a rectangular box is generated with each fixation point as the center.

[0096] Next, we calculated the computational speed and accuracy of different methods. Computational speed is expressed in FPS (Frames Per Second), which measures the number of frames (images) the model can process (or detect) per second, measured in frames per second. Accuracy refers to the percentage of objects within the selected candidate boxes, expressed as the ratio of the total area of ​​the objects in the image to the total area of ​​the candidate boxes.

[0097] In summary, combining accuracy, precision, and computational speed, we determined that input image size and minimum distance are the optimal parameters for the candidate box selection model. The fusion method outperforms the basic method in comprehensive capabilities, and the saliency-dominated method is more effective in terms of computational speed and accuracy.

[0098] The following describes the target detection test using a dual-channel vision mechanism candidate box selection model.

[0099] First, a dual-channel vision mechanism candidate box selection model is used to generate candidate boxes. The image is then cropped according to the candidate boxes and fed into the trained Faster-RCNN for object recognition, outputting the object's bounding box and probability. The test parameters are set as follows: input image size of 500*500, minimum distance of 50, and recognition confidence greater than 70% is considered correct. The remote sensing object detection dataset is used for testing, with the detected targets being the dataset's labeled boxes, i.e., the ground truth.

[0100] Furthermore, the recall and precision are used to evaluate the detection results. The recall rate represents the proportion of correct detections among all targets, and the calculation formula is:

[0101]

[0102] Where TP represents the number of detection boxes with an IoU greater than the threshold of 0.5, and all ground truths represents the number of all targets.

[0103] The precision rate indicates the correct detection ratio among all predicted boxes, and the calculation formula is:

[0104]

[0105] Where TP represents the number of detection boxes with an IoU greater than the threshold 0.5, and all ground truths represents the number of all predicted boxes.

[0106] Based on the above research results, the embodiment of this application proposes a candidate box selection model based on a dual-channel visual mechanism. This model is divided into three modules:

[0107] (1) A top-down visual attention mechanism module, which uses experimental data to train the deep learning model MSI-Net to predict the attention saliency of human search for aircraft;

[0108] (2) A bottom-up visual attention mechanism module that uses the visual saliency calculation model Itti to calculate the visual saliency of the image;

[0109] (3) A dual-channel visual attention mechanism fusion module proposes a visual saliency-dominated method and an attention saliency-dominated method to generate candidate boxes.

[0110] Finally, the embodiment of the present application studies the selection performance of the candidate box selection model under different parameter settings through ablation experiments, and determines the optimal parameters.

[0111] The target detection method proposed in the embodiment of this application adaptively generates candidate frames based on the position and size of the target, breaking through the fixed size and fixed step size problems of traditional candidate frame selection and being able to select target candidate frames on remote sensing images of different scales. At the same time, compared with the Faster-RCNN method that generates tens of thousands of candidate frames, the embodiment of this application only needs to generate a few to dozens of candidate frames to complete the prediction of the target position and size, achieving rapid and accurate selection of candidate frames, improving target detection accuracy and speed.

[0112] Based on the above model, preliminary target detection is completed. However, if there are many misclassifications in the detection results, in order to solve the problem of target misclassification, the embodiment of the present application divides the objects to be identified in the candidate box into targets and non-targets, analyzes the researchers' cognitive differences between the two objects, determines the features that affect the classification accuracy, constructs a difficult-to-distinguish sample set based on the research results, and conducts intensive training on the recognition model to effectively improve the recognition accuracy.

[0113] Based on the research objectives, we selected researchers with good data collection and excellent experimental performance for analysis to build an efficient target detection model. The target detection accuracy of the researchers was calculated by counting whether their gaze points fell within the target range. The specific process is as follows:

[0114] 1. Gaze point aggregation. When the straight-line distance between two temporally adjacent gaze points does not exceed 1 degree from the center of the visual field, the two gaze points are merged, their coordinates are averaged, and the gaze durations are accumulated. Each gaze point must have a duration of at least 100ms. Gaze points with a duration of less than 100ms are discarded. The preprocessed gaze set is aggregated into gaze points according to the above rules.

[0115] 2. Determine whether the researcher has found the target. Using the labeled data from the remote sensing image dataset, calculate whether the gaze point falls within the verification box and calculate the gaze duration. If it exceeds 100ms, the target is considered to have been identified.

[0116] 3. Count the number of targets on all stimulus materials and the number of recognitions by researchers. At the same time, the embodiment of the present application uses EEG data to evaluate the cognitive load of researchers during the recognition process. Cognitive load generally refers to the ratio of the brain's information processing capacity to task requirements under working conditions. Cognitive load is affected by many other factors such as task requirements, time pressure, personal ability, degree of effort, task benefits, etc. Excessive cognitive load may cause rapid fatigue of the brain, trigger a stress response, increase the possibility of human error, cause errors in information acquisition and analysis, and make wrong decisions, which is an important reason for the decline in work efficiency.

[0117] Current research shows that the power spectral density of different frequency bands of EEG can effectively measure cognitive load. Gevins et al. used a matching task to test the cognitive load of researchers. The results showed that the amplitude of theta waves in the middle of the frontal lobe increased with the increase of memory burden, and the amplitude of alpha waves in the center of the parietal lobe decreased with the increase of working memory load. Keskin et al. measured the cognitive burden of map reading for experts and beginners. The results showed that at all difficulty levels of experts and novices, the power spectral density of theta waves increased, while the alpha waves decreased. In summary, the cognitive load of researchers is related to theta waves and alpha waves. Based on this, the index based on dual-channel EEG in the embodiment of the present application is used to evaluate the cognitive burden of the brain during the task. The calculation formula of the dual-channel EEG cognitive burden evaluation index is:

[0118]

[0119] Where θ Fz is the theta wave intensity of EEG at Fz channel, α Pz It is the intensity of the alpha wave in the Pz channel of the EEG. The larger the index value, the higher the cognitive load of the researcher. This embodiment of the application uses this index to measure the cognitive load of the researcher during the experiment.

[0120] To address the problem of existing models misclassifying targets, we used EEG data to analyze researchers' brain activity when they fixated on targets and non-targets. We extracted targets and non-targets from the salient area of ​​attention, and calculated ERPs using the fixation point as a marker. We used the onset of fixation within the candidate frame as the event start and analyzed the researchers' brain activity when they fixed on targets and non-targets. Based on the researchers' target recognition cognitive mechanisms, we constructed a set of difficult-to-distinguish samples, intensively trained the recognition model, and constructed a complete target detection model.

[0121] Based on ERP analysis, non-target objects in the candidate frames that are easily misclassified often resemble targets in visual features such as color and shape. However, due to a lack of training samples for these types of objects, the model cannot distinguish them from targets during training and may mistake them for airplanes. Therefore, the experimental material was selected from 500 images, from which over 1,000 difficult-to-classify samples were labeled. The primary target was objects with similar colors and shapes to airplanes, such as covered bridges, white cars, and house roofs.

[0122] The difficult-to-distinguish samples and target samples are input into the model for training at the same time, and the trained model is used for detection. In order to comprehensively consider the precision and recall rates, the experiment adds AP (Average Precision) and F1 scores as model evaluation indicators. AP is the average precision, which refers to the area under the Precision-Recall curve. Specifically, AP first calculates the precision (Precision) and recall (Recall) at each confidence level, and then draws the Precision-Recall curve with Recall as x and Precision as y, and then integrates the Precision-Recall curve to obtain the area under the curve, namely AP. The AP value ranges from 0 to 1. The larger the value, the better the detection result. In actual calculations, the value of Precision takes the maximum Precision value to the right of the point, and the calculation formula is:

[0123]

[0124] Where M is the number of Recall, pre i is the Precision value corresponding to Recall.

[0125] The F1 score is the harmonic mean of precision and recall. It is mainly used to evaluate the performance of classifiers, but can also be used to evaluate the performance of models in object detection. The value range is between 0 and 1, and the larger the value, the better the model performance. The calculation formula is as follows:

[0126]

[0127] Where P is the precision rate and R is the accuracy rate.

[0128] Experiments show that after training on difficult-to-classify samples, the model's recall and precision rates have improved, and the overall performance has been significantly improved.

[0129] This example uses EEG data to analyze the EFRP when researchers gaze at targets and non-targets, studying the cognitive mechanisms by which researchers judge and identify targets. This provides a theoretical basis for constructing a dataset of difficult-to-separate samples. The study shows that researchers' target recognition follows the following cognitive patterns:

[0130] (1) The color, shape, and other visual features of targets and non-targets are similar, and the primary visual cortex in the occipital lobe of researchers has similar neural responses to both, making it impossible to distinguish between targets and non-targets;

[0131] (2) 200ms after the primary visual cortex completes the primary processing, the frontal lobe discriminates whether the gaze object is a target;

[0132] (3) 300ms after the primary visual cortex completes primary processing, the temporal lobe extracts the semantic information of the gazed object, and the researchers complete target recognition.

[0133] Based on the above research results, the embodiment of this application selected non-target objects of similar color and shape to construct a difficult-to-distinguish sample dataset and trained the target detection model. The test results showed that the precision rate increased by 17% and the number of misclassified targets was greatly reduced. To test the effectiveness of the model, the embodiment of this application conducted an untrained test on the dataset, with the accuracy rates of . The results show that the multi-scale remote sensing image target detection model based on the visual attention mechanism improves by an average of 26.56% compared to Faster-RCNN.

[0134] Remote sensing target detection is an important part of remote sensing image interpretation. Its automation level directly affects the application of remote sensing data and has important research value. In view of the problem that remote sensing images have large scale changes and traditional target detection models are difficult to effectively identify targets, starting from the human visual cognitive mechanism, by designing and implementing eye movement-EEG target detection cognitive experiments, we explore the multi-scale target retrieval cognitive laws of professional interpreters and study the target search cognitive mechanism of the subjects; based on the target search cognitive law, we propose a candidate box selection model based on a dual-channel visual attention mechanism, which realizes the rapid and accurate selection of candidate boxes; by jointly analyzing eye movement and EEG data, we study the target recognition cognitive mechanism of the subjects, construct a difficult-to-separate sample data set based on the target recognition mechanism, improve the recognition accuracy of the detection model, and realize target detection of multi-scale remote sensing images. Through the above research, the main research conclusions of the method provided in the embodiment of this application are as follows:

[0135] 1. By analyzing researchers' scanning and fixation behaviors, we investigated the factors influencing their multi-scale target search. The results showed that: ① Researchers' target search was driven by both top-down task-driven and bottom-up visual stimulation, with the top-down attention mechanism being dominant. ② The greater the number of targets in a remote sensing image and the smaller their scale, the more focused researchers' attention was on the target. The larger the image scale, the more frequent the search. Furthermore, researchers were able to locate the suspected target in their peripheral visual field during fixation, providing guidance for their scans. ③ Color was the primary feature researchers focused on when searching for a target, followed by edge features.

[0136] 2. To address the large scale variations in remote sensing imagery, researchers, building on the cognitive laws of target detection, studied the relationship between visual saliency, target semantic saliency, and attentional saliency, and proposed a candidate box selection model based on a dual-channel visual attention mechanism. The results showed that: ① Neither attentional saliency nor visual saliency can independently predict the complete semantic saliency of a target, but when combined, they can predict all target locations. ② Using deep learning and computational models to calculate attentional saliency and visual saliency, simulating the human "top-down" and "bottom-up" visual attention mechanisms, a dual-channel visual attention fusion method was proposed, achieving precise candidate box selection on multi-scale remote sensing imagery with an accuracy of 70.7%.

[0137] 3. By studying the neural activity of researchers when identifying targets, we explored the characteristics of difficult-to-distinguish samples, constructed a dataset of difficult-to-distinguish samples, and improved the model's recognition accuracy. Research results: ① When researchers looked at targets and non-targets, the occipital lobe neural activity was similar, while there were significant differences in the frontal and temporal lobes, indicating that the primary visual features of the targets and non-targets, such as color and shape, were similar. ② Non-target objects of similar color and shape were selected to construct a dataset of difficult-to-distinguish samples for model training. Test results showed that the precision rate increased by 17%, and the number of misclassified targets was greatly reduced. ③ In a non-training test on the dataset, the model of the embodiment of the present application was 26.56% higher than Faster-RCNN on average.

[0138] In summary, the present embodiment provides a method for remote sensing image target detection. This method uses a dual-channel vision mechanism candidate frame selection model to generate target candidate frames for remote sensing images, then crops the remote sensing image according to the target candidate frames. The cropped image is then input into a trained target detection model for target recognition, outputting the target's bounding box and probability. This method is capable of selecting target candidate frames for remote sensing images of varying scales, efficiently predicting the target's position and size, and enhancing the intelligent detection capabilities of multi-scale remote sensing images.

[0139] Based on the same technical concept, the present application also provides a remote sensing image target detection system, such as Figure 4 As shown, the system includes:

[0140] A target candidate frame module 401 is configured to generate a target candidate frame for a remote sensing image using a dual-channel vision mechanism candidate frame selection model, and crop the remote sensing image according to the target candidate frame;

[0141] The target recognition module 402 is used to input the cropped image into the trained target detection model for target recognition and output the target's bounding box and probability.

[0142] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Figure 5 , which shows a schematic diagram of an electronic device provided in some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program executable on the processor 200. When the processor 200 executes the computer program, it executes the method provided in any of the aforementioned embodiments of the present application.

[0143] The memory 201 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element and at least one other network element are connected via at least one physical port 203 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0144] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. The processor 200 executes the programs upon receiving execution instructions. The methods disclosed in any of the aforementioned embodiments of the present application may be applied to or implemented by the processor 200.

[0145] The processor 200 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 200 or by software instructions. The above processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 201 , and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0146] The electronic device provided in the embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0147] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 6 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided by any of the aforementioned embodiments is executed.

[0148] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0149] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0150] It should be noted that:

[0151] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other device. Various general-purpose devices may also be used in conjunction with the teachings herein. Based on the above description, it is apparent that the structure required for constructing such devices is suitable. In addition, the present application is not directed to any specific programming language. It should be understood that various programming languages ​​may be utilized to implement the present application described herein, and the above description of specific languages ​​is provided for the purpose of disclosing the best mode of implementation of the present application.

[0152] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0153] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in fewer than all the features of the individual embodiments disclosed above. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim itself serving as a separate embodiment of the present application.

[0154] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0155] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0156] The various component embodiments of the present application can be implemented in hardware, or implemented in a software module running on one or more processors, or implemented in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the creation device of the virtual machine according to an embodiment of the present application. The application can also be implemented as a part or all of the equipment or device program (for example, computer program and computer program product) for performing the method described herein. Such a program realizing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0157] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0158] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A remote sensing image target detection method, characterized in that: The method comprises: A dual-channel vision mechanism candidate frame selection model is used to generate a target candidate frame for the remote sensing image, and the remote sensing image is cropped according to the target candidate frame; The cropped image is input into the trained object detection model for object recognition, and the bounding box and probability of the object are output; The dual-channel visual mechanism candidate frame selection model includes a visual saliency calculation model and an attention saliency calculation model; the remote sensing image is used to generate a target candidate frame using the dual-channel visual mechanism candidate frame selection model, including: determining the position of the target candidate frame based on the attention saliency calculation model for the remote sensing image; determining the size of the target based on the visual saliency calculation model for the remote sensing image; and adaptively generating the target candidate frame according to the position of the target candidate frame and the size of the target; wherein, the remote sensing image is determined based on the visual saliency calculation model to determine the size of the target, and the remote sensing image is determined based on the attention saliency calculation model to determine the size of the target. The method comprises the following steps: converting the remote sensing image into a visual saliency map and an attention saliency map; converting a salient area in the visual saliency map into a visual saliency object using a grayscale segmentation threshold; extracting a fixation point from the attention saliency map using a local maximum method; selecting a visual saliency object that coincides with the fixation point to generate a bounding rectangle for each visual saliency object, and determining the size of the target based on the bounding rectangle; considering all fixation points as target positions, and if the fixation point is within the visual saliency object, determining the position of the target candidate frame based on the visual saliency object; and if the fixation point is not within the visual saliency object, determining the position of the target candidate frame with the fixation point as the center.

2. The method according to claim 1, wherein The attention saliency calculation model is constructed according to the following steps: Pre-training an attention saliency calculation model using a remote sensing image eye movement dataset, wherein the remote sensing image eye movement dataset includes geospatial images of maps, remote sensing images, and street scenes; wherein the attention saliency calculation model is loaded with pre-trained weights of a VGG-16 convolutional neural network, and a data augmentation method is used to expand the size of the remote sensing image eye movement dataset; The eye movement data collected in the experiment and the corresponding remote sensing images are used to train and test the attention saliency calculation model.

3. The method according to claim 1, wherein The target detection model is constructed according to the following steps: Select remote sensing images of various scales as stimulus materials to obtain eye movement and brain neural data when users perform visual target detection; Analyze the user's eye movement patterns and neural responses when identifying the target object based on eye movement and brain neural data, extract the target objects that are easy to make mistakes and difficult to distinguish, and construct a difficult-to-distinguish sample set; The Faster-RCNN model is trained based on the difficult-to-classify sample set to obtain the target detection model.

4. The method according to claim 1, wherein The method further comprises: Calculate recall and precision based on the bounding box and probability of the target; The object detection model is optimized based on recall and precision.

5. A remote sensing image target detection system, characterized in that: The system comprises: The target candidate frame module is used to generate a target candidate frame for a remote sensing image using a dual-channel visual mechanism candidate frame selection model, and to crop the remote sensing image according to the target candidate frame; the dual-channel visual mechanism candidate frame selection model includes a visual saliency calculation model and an attention saliency calculation model; the remote sensing image is generated using the dual-channel visual mechanism candidate frame selection model to generate a target candidate frame, including: determining the position of the target candidate frame for the remote sensing image based on the attention saliency calculation model; determining the size of the target for the remote sensing image based on the visual saliency calculation model; adaptively generating the target candidate frame according to the position of the target candidate frame and the size of the target; wherein, the remote sensing image is determined based on the visual saliency calculation model to determine the target candidate frame. The method comprises the following steps: converting the remote sensing image into a visual saliency map and an attention saliency map; converting a salient area in the visual saliency map into a visual saliency object using a grayscale segmentation threshold; extracting a fixation point from the attention saliency map using a local maximum method; selecting a visual saliency object that coincides with the fixation point to generate a bounding rectangle of each visual saliency object, and determining the size of the target according to the bounding rectangle; considering all fixation points as target positions, and if the fixation point is within the visual saliency object, determining the position of the target candidate frame according to the visual saliency object; and if the fixation point is not within the visual saliency object, determining the position of the target candidate frame with the fixation point as the center. The target recognition module is used to input the cropped image into the trained target detection model for target recognition and output the target's bounding box and probability.

6. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 4.

Images