Method, device, equipment and medium for screening regions of interest in remote sensing images
By combining object detection and region significance detection, screening and fusing regions of interest in remote sensing images, the problem of low screening efficiency of regions of interest in remote sensing images is solved, and efficient data compression and bandwidth optimization are achieved.
Patent Information
- Application Number
- CN202210504121.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-05-10
AI Technical Summary
The prior art is difficult to quickly and accurately screen out the regions of interest in remote sensing images, resulting in low data transmission efficiency and inability to meet real-time processing requirements.
A combination of object detection and regional significance detection is adopted to generate regions of interest, and important regions are screened out through fusion and sorting, achieving significant data compression and bandwidth reduction.
It realizes the rapid and accurate extraction of the region of interest from large-format remote sensing images, realizes 30-50 times data compression, greatly reduces bandwidth requirements, and meets real-time processing requirements.
Smart Images

Figure CN115082668B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method, device, equipment and medium for screening regions of interest in remote sensing images. Background Art
[0002] Remote sensing images, also known as surface images, are high-altitude images of the earth's surface obtained by high-altitude aircraft or near-earth satellites through cameras or other sensors, with a resolution of up to about 0.1 meters. Remote sensing images are large in size and usually have a large amount of data. For example, a 18-level remote sensing image usually has a data volume of about 2G to 20G. Under current network bandwidth conditions, it usually takes 6 to 60 minutes to transmit images of this magnitude, which is unacceptable in scenarios with real-time processing requirements.
[0003] Therefore, how to extract important areas from large-format remote sensing images through regional intelligent screening technology is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention
[0004] The embodiments of the present application provide a method, device, equipment and medium for screening regions of interest in remote sensing images. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.
[0005] In a first aspect, an embodiment of the present application provides a method for screening a region of interest in a remote sensing image, comprising:
[0006] Detecting a target of interest in the remote sensing image according to a target detection algorithm, generating a target bounding box, and obtaining one or more first regions of interest according to the target bounding box;
[0007] Detecting a salient region in the remote sensing image according to a regional saliency detection algorithm to obtain one or more second regions of interest;
[0008] Merging the first region of interest and the second region of interest whose similarity is greater than a preset threshold to obtain a fused region of interest;
[0009] The fused region of interest and the unfused first region of interest and the second region of interest are ranked in relative importance, and the selected region of interest is obtained according to the ranking result.
[0010] In some optional embodiments, obtaining one or more first regions of interest according to the target bounding box includes:
[0011] Enlarging the target bounding box according to a preset ratio to obtain an enlarged target bounding box;
[0012] Determine whether the enlarged target bounding boxes intersect, and merge the intersecting enlarged target bounding boxes;
[0013] Determine whether the side length of the enlarged target bounding box is greater than a preset threshold, and determine whether the distance between two enlarged target bounding boxes whose side lengths are greater than the preset threshold is less than the preset threshold;
[0014] If the side length of the enlarged target bounding box is greater than a preset threshold, and the distance between the two enlarged target bounding boxes whose side lengths are greater than the preset threshold is less than the preset threshold, the two enlarged target bounding boxes are merged;
[0015] One or more first regions of interest are generated according to the merged target bounding boxes.
[0016] In some optional embodiments, detecting a salient region in a remote sensing image according to a regional saliency detection algorithm to obtain one or more second regions of interest includes:
[0017] Inputting the remote sensing image into the regional saliency detection model to obtain one or more detected irregular saliency maps;
[0018] The pixels in the irregular saliency map that are greater than or equal to the segmentation threshold are retained, and the pixels that are less than the segmentation threshold are removed to obtain a regular segmentation boundary;
[0019] One or more second regions of interest are obtained according to the regular segmentation boundaries.
[0020] In some optional embodiments, the remote sensing image is input into a regional saliency detection model to obtain one or more detected irregular saliency maps, including:
[0021] The remote sensing image is input into the regional saliency detection model, and the remote sensing image is reduced by bilinear interpolation and random cropping to obtain a reduced remote sensing image;
[0022] The reduced remote sensing image is input into the fully convolutional neural network to obtain the predicted rough saliency map;
[0023] The rough saliency map and the remote sensing image to be detected are input into the residual refinement module to obtain one or more detected irregular saliency maps.
[0024] In some optional embodiments, the first region of interest and the second region of interest whose similarity is greater than a preset threshold are merged to obtain a merged region of interest, including:
[0025] Calculate the intersection-and-union ratio of the first region of interest and the second region of interest, and obtain the similarity according to the intersection-and-union ratio;
[0026] The first region of interest and the second region of interest whose similarity is greater than a preset threshold are fused, and the bounding box of the second region of interest is used as the bounding box of the fused region of interest.
[0027] In some optional embodiments, the fused region of interest and the unfused first region of interest and the second region of interest are ranked in relative importance, and the selected region of interest is obtained according to the ranking result, including:
[0028] According to the target category in the region of interest to be screened, obtaining the fused region of interest and the unfused first region of interest and the region of interest containing the same target category in the second region of interest;
[0029] Calculate the confidence of the region of interest containing the same target category;
[0030] The regions of interest containing the same target category are sorted from large to small according to the confidence level, and a preset number of regions of interest at the front are used as the screened regions of interest.
[0031] In some optional embodiments, calculating the confidence of the regions of interest containing the same target category includes:
[0032] The confidence level is calculated according to the following formula:
[0033]
[0034] Among them, W1, W2, W3, and W4 represent weight parameters, PN roi Represents the foreground pixel in the region of saliency detection, PN ori represents the pixel of the detected image, T s Represents the confidence threshold in saliency detection, score i represents the confidence of the i-th target in target detection, S roi Indicates the number of targets in the bounding box in target detection, S ori Represents the area of the bounding box in target detection.
[0035] In a second aspect, an embodiment of the present application provides a device for screening a region of interest in a remote sensing image, comprising:
[0036] A first detection module, used to detect a target of interest in the remote sensing image according to a target detection algorithm, and generate a target bounding box, and obtain one or more first regions of interest according to the target bounding box;
[0037] A second detection module, used for detecting a salient region in the remote sensing image according to a regional saliency detection algorithm, and obtaining one or more second regions of interest;
[0038] A fusion module, used for fusing the first region of interest and the second region of interest whose similarity is greater than a preset threshold to obtain a fused region of interest;
[0039] The sorting and screening module is used to sort the relative importance of the fused regions of interest and the unfused first regions of interest and second regions of interest, and obtain the screened regions of interest according to the sorting results.
[0040] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing program instructions, wherein the processor is configured to execute the method for screening regions of interest in remote sensing images provided in the above embodiment when executing the program instructions.
[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having computer-readable instructions stored thereon, and the computer-readable instructions are executed by a processor to implement a method for screening regions of interest in a remote sensing image provided in the above embodiment.
[0042] The technical solution provided by the embodiments of the present application may have the following beneficial effects:
[0043] According to a method for screening regions of interest in remote sensing images provided by an embodiment of the present application, for large-format remote sensing images, a regional intelligent screening technology based on target detection is first proposed, and a large-scale region of interest is generated based on the target detection results. Secondly, in order to solve the problem of insufficient reliability of target detection technology, a regional intelligent screening technology based on regional significance detection is proposed, which detects significant regions from large-format remote sensing images and generates regular rectangular slices. Thirdly, the regions based on target detection and the regions based on regional significance detection are fused to obtain the final candidate regions. Finally, the candidate regions are sorted by importance, and the selected regions of interest are obtained according to the sorting results. This method can accurately and quickly extract regions of interest from large-format remote sensing images and transmit them, achieving 30-50 times compression without losing important information, greatly reducing the bandwidth requirements.
[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0046] Figure 1 is a flow chart of a method for screening a region of interest in a remote sensing image according to an exemplary embodiment;
[0047] Figure 2 is a schematic diagram of a method for merging target bounding boxes according to an exemplary embodiment;
[0048] Figure 3 is a schematic diagram of a method for merging target bounding boxes according to an exemplary embodiment;
[0049] Figure 4 is a schematic diagram of another method for merging target bounding boxes according to an exemplary embodiment;
[0050] Figure 5 is a schematic diagram showing a merged first region of interest according to an exemplary embodiment;
[0051] Figure 6 is a structural schematic diagram of a saliency detection model according to an exemplary embodiment;
[0052] Figure 7 is a network diagram of a saliency detection model according to an exemplary embodiment;
[0053] Figure 8 is a schematic structural diagram of a device for screening regions of interest in remote sensing images according to an exemplary embodiment;
[0054] Fig. 9 is a schematic structural diagram of an electronic device according to an exemplary embodiment;
[0055] Fig.10 It is a schematic diagram of a computer storage medium according to an exemplary embodiment. DETAILED DESCRIPTION
[0056] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
[0057] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of systems and methods consistent with some aspects of the present invention as detailed in the attached claims.
[0059] At present, a commonly used solution is to use target detection technology. Target detection technology can find the location of the target in a given image and mark the category of the target. It is usually used to locate the target area of interest. Given a picture as input, target detection technology first extracts candidate boxes from the image, then extracts features from the candidate boxes and uses a classifier to determine whether it belongs to the target or the background. Finally, the non-maximum suppression algorithm is used to merge the candidate boxes to obtain the final target detection result.
[0060] However, although target detection technology can detect target instances, it cannot locate larger areas. For example, target detection technology can locate targets such as aircraft or ships in remote sensing images, but it cannot locate the entire airport area or port area. In addition, although most existing deep learning-based target detection methods have achieved good detection results in natural images, compared with natural images, remote sensing images have large differences in target angle rotation, scale changes, complexity, and messy backgrounds, making target detection technology less reliable on remote sensing images.
[0061] Based on this, an embodiment of the present application provides a method for screening areas of interest in remote sensing images, which mainly solves the problem of intelligent screening of important areas of large-format remote sensing images, thereby transmitting the important areas of the collected remote sensing images in blocks, improving bandwidth utilization efficiency and coding compression efficiency, and realizing real-time coarse screening of scenes and targets of interest.
[0062] This application first proposes a regional intelligent screening technology based on target detection, and generates a large range of regions of interest based on the target detection results. Secondly, in order to solve the problem of insufficient reliability of target detection technology, a regional intelligent screening technology based on regional saliency detection is proposed to detect salient regions from large-format remote sensing images and generate regular rectangular slices. Thirdly, the regions obtained by regional intelligent screening based on target detection and the regions obtained by regional intelligent screening based on regional saliency detection are fused to obtain the final candidate regions. Finally, the candidate regions are sorted by importance.
[0063] The following is a detailed description of the method for screening regions of interest in remote sensing images provided by the embodiments of the present application in conjunction with the accompanying drawings. Figure 1 , the method specifically includes the following steps.
[0064] S101 detects a target of interest in a remote sensing image according to a target detection algorithm, generates a target bounding box, and obtains one or more first regions of interest according to the target bounding box.
[0065] Among them, the target detection algorithm can be some conventional target detection algorithms in the prior art, for example, an algorithm based on region detection, such as R-CNN, Fast R-CNN, Faster R-CNN, etc. Detection algorithms based on target regression, such as YOLO, SSD, RetinaNet, EfficientDet, etc. The embodiment of the present application does not limit the specific target detection algorithm. In addition, the target of interest in the remote sensing image to be detected can be an airplane, a ship, a car, etc., so as to locate the airport area, the port area or the parking lot area.
[0066] Specifically, firstly, a remote sensing image to be detected is obtained, and the remote sensing image to be detected is input into a target detection algorithm to generate multiple target bounding boxes. In the embodiment of the present application, the target of interest may be an airplane, a ship, a car, etc., so that the target detection algorithm is used to detect the airplane target, the ship target, and the vehicle target in the remote sensing image, and generate a target bounding box.
[0067] Further, one or more first regions of interest are obtained according to the target bounding box. First, the target bounding box is enlarged according to a preset ratio to obtain an enlarged target bounding box. For example, the obtained target bounding box is enlarged 10 times in equal proportion. The specific enlargement ratio can be set according to the actual situation. Then, the enlarged target bounding boxes are merged. First, it is determined whether the enlarged target bounding boxes intersect, and the intersecting enlarged target bounding boxes are merged. It is determined whether the side length of the enlarged target bounding box is greater than a preset threshold, and it is determined whether the distance between the two enlarged target bounding boxes with a side length greater than the preset threshold is less than the preset threshold; if the side length of the enlarged target bounding box is greater than the preset threshold, and the distance between the two enlarged target bounding boxes with a side length greater than the preset threshold is less than the preset threshold, then the two enlarged target bounding boxes are merged. For example, although the two enlarged target bounding boxes do not intersect, they are relatively large and close to each other. The target bounding boxes in this case are also merged until they can no longer be merged to obtain one or more merged first regions of interest.
[0068] Remote sensing images are usually top-down views of objects, and objects usually do not block each other. Object detection technology obtains non-intersecting areas in remote sensing images, but these areas do not contain any contextual information and scene information, so they are not sufficient as output results. After the region merging technology, the loss of corresponding information can be minimized as much as possible. For example, the merged area of airplanes supplements the airport area, and the merged area of ships supplements the port area.
[0069] Figure 2 is a schematic diagram of a method for merging target bounding boxes according to an exemplary embodiment; Figure 2 As shown, first, in the (result, result) picture below, the target bounding box is obtained by the target detection algorithm, and then, in the (process, process) picture above, according to the ratio of the plane to the airport, the ratio of the ship to the port, etc., the bounding box of the plane and other objects is finally enlarged by 10 times, and then (calcu inter merge, calculate intersection merge, calculate intersection merge), if the bounding boxes intersect with each other, or the bounding boxes are relatively large and close to each other, they are considered to belong to the same object in the same area, and the area is merged. The merging result is shown on the right side of the (process, process) picture above, and then due to (border limit, boundary limit), the region of interest in the remote sensing image obtained after the final merger is shown on the right side of the (result, result) picture below.
[0070] Figure 3 is a schematic diagram showing a merging of target bounding boxes according to an exemplary embodiment. Figure 3 As shown, the bounding box of the aircraft is first obtained through the target detection algorithm, such as Figure 3 As shown on the left, they are then merged. If the bounding boxes intersect with each other, or the bounding boxes are relatively large and close to each other, they are considered to be objects in the same area, and the areas are merged to obtain the merged area bounding box, as shown in Figure 3 Shown on the right.
[0071] At present, the merging of vertical detection frames has been completed. However, after analyzing the detection results of vertical detection frames, it is found that some uninteresting areas are included due to the irregular shape of the airport, and a considerable part of the interesting areas are discarded. Therefore, it is hoped that on the basis of the original detection, the detection frame can contain angle information to form a rotation frame, so as to obtain a more accurate area selection result. The rotation frame is formed according to the coordinate information of each target bounding box, such as Figure 4 As shown in the figure, the optimized rotating frame can contain more information about the region of interest than the vertical frame and is closer to the actual region range.
[0072] In small-scale remote sensing images where the target area occupies the absolute majority of the area, good results can be obtained through regional merging technology. However, in actual processes, such as the positioning of airport areas, after the aircraft is detected and the regions are merged, the context information is not very good. There are many scattered areas, including false positives due to misdetection, the situation when the aircraft is flying in the air, etc., among which false positives account for a relatively large number.
[0073] Optionally, a post-processing method is adopted to suppress the corresponding situation. This method can be widely used in various area detection scenarios such as airports, ports, parking lots, etc., which can significantly improve the accuracy while reducing the interference of useless information.
[0074] First, airplanes are usually isolated when flying on routes, and there is only one airplane in the corresponding bounding box after region merging, which is completely different from the density of airplanes on the apron at the airport. The same is true for ships. Ships are densely packed in ports, while ships on the sea need to be far apart to ensure navigation safety. Therefore, the first step in post-processing is to delete the bounding box that contains only one object of interest. When an airplane is misjudged, the confidence level is usually low, often because of similar textures or similar color distributions. In this case, the accuracy can be greatly improved by increasing the corresponding confidence threshold. Finally, the apron at the airport is large, and the airplanes are not evenly distributed, resulting in discontinuous regional merging results. Therefore, when merging, detection boxes that are not far apart should be considered to belong to the same area even if they do not intersect with each other, so as to avoid missing important information. The final merged area after post-processing is as follows Figure 5 The figure shows a schematic diagram of the first region of interest after merging. This method can be reused in a variety of scenes of interest, thereby improving the accuracy and recall rate of the target detection module and providing important support for the subsequent region importance ranking.
[0075] S102 detects salient regions in the remote sensing image according to a regional saliency detection algorithm to obtain one or more second regions of interest.
[0076] Specifically, the remote sensing image is first input into the regional saliency detection model to obtain one or more irregular saliency maps. Figure 6 As shown, it includes a bilinear difference random cropping module, a full convolutional neural network module and a residual refinement module, inputs the remote sensing image into the regional saliency detection model, reduces the remote sensing image by bilinear difference and random cropping, and obtains the reduced remote sensing image; inputs the reduced remote sensing image into the full convolutional neural network to obtain a predicted rough saliency map; inputs the rough saliency map and the remote sensing image to be detected into the residual refinement module to obtain one or more detected irregular saliency maps.
[0077] The saliency detection algorithm proposed in the embodiment of the present application adopts a coarse-to-fine structure. During the data loading stage, the large-format remote sensing image is first reduced to a size of 256×256 by bilinear interpolation, and then further reduced to 224×224 by random cropping. Next, a rough saliency map is predicted through an image prediction module based on a fully convolutional neural network. The rough saliency map is sent to the residual refinement module to further optimize the prediction results.
[0078] After the image is reduced, the image prediction module and the residual refinement module are mainly used for regional prediction, such as Figure 7 As shown in Figure 1. The main task of the image prediction module is to preliminarily predict the saliency map. The module is designed as an encoder-decoder network similar to U-Net. The encoder includes 6 basic residual modules. The first 4 residual modules come from the first 4 residual modules of ResNet-34 pre-trained on ImageNet. ResNet-34 is used to extract the features of the input image. In order to obtain a larger receptive field, the last two residual modules use a convolutional layer with 512 3×3 dilated convolutions. Similar to U-Net, in order to capture global information, connections are made between the encoding stage and decoding stage modules of the same size. The structure of the decoder is similar to that of the encoder. Each stage consists of a module consisting of three convolutional layers, batch normalization, and activation functions. In addition to being connected to the next decoding stage, each decoding stage is also connected to an output layer, which consists of a convolutional layer, a bilinear interpolation function, and a sigmoid activation function. The convolutional layer is further decoded, the bilinear interpolation function performs upsampling to restore the original resolution, and the sigmoid function limits the predicted value to between 0 and 1. The decoder contains seven decoding stages, so it will output seven saliency maps of the same size, each of which participates in the calculation of the loss function. The last saliency map has the highest accuracy, so it is sent to the residual refinement stage to further optimize the prediction results.
[0079] It is difficult to accurately predict the boundary using only the prediction module, so a residual refinement module is designed after the prediction module to optimize the overall salient target and the boundary area. The residual refinement module takes the rough saliency map and the original image provided by the prediction module as input, calculates the residual between the rough saliency map and the true value through a convolutional neural network, and superimposes the residual and the rough saliency map to obtain the final result. The structure of the residual refinement module is similar to that of the prediction module, but the task of the residual refinement module is simpler than that of the prediction module, so the structure is also simpler. The residual refinement module consists of a 4-stage encoder and a 4-stage decoder, and is also connected between the encoding stage and the decoding stage of the same scale. Unlike the prediction module, each stage of the refinement module consists of only one convolution layer, batch normalization and activation function. The number of convolution kernels in each stage is set to 64, and the entire module has only one output.
[0080] In addition to the design of the network structure, the design of the loss function is also very critical. In order to better detect salient targets, the loss function consists of three parts, including the cross entropy loss function, the structural similarity loss function, and the intersection-over-union loss function. Among them, the cross entropy loss function is widely used in saliency detection of binary classification tasks. In this model, it is mainly hoped that the background can be suppressed and the foreground can be highlighted through pixel-level restrictions. In formula (1), G(r,c) represents the true value of the pixel, and S(r,c) represents the model prediction value:
[0081]
[0082] The structural similarity loss function is used to constrain the structure and boundaries. Under the same prediction probability, the edge has a greater loss than the internal pixel, thus obtaining a stronger optimization. The calculation formula is as follows. x represents the N×N block predicted by the model, y represents the N×N block in the corresponding true value, μx, μy, σx, σy represent the mean and standard deviation of x and y respectively, which are used to represent the structural information of the segmentation result. C1=0.012 and C2=0.032 are used to avoid division by zero.
[0083]
[0084] The intersection-over-union loss function is similar to that in target detection, but because it is extended to the pixel level, the amount of calculation is relatively large. This loss forces the model to learn sufficient translation invariance to ensure that the loss is reduced.
[0085]
[0086] Finally, all the loss calculation formulas are as follows, so that the saliency model reaches a convergence state in three aspects: foreground-background distinction, saliency subject boundary, and saliency subject itself.
[0087] L = l bce +l ssim +l iou #(4)
[0088] Furthermore, pixels in the irregular saliency map that are greater than or equal to the segmentation threshold are retained, and pixels that are less than the segmentation threshold are removed, so as to obtain a regular segmentation boundary.
[0089] The result of regional saliency detection is an irregular grayscale pixel-level image mask, which is inconsistent with the final required regular slice. First, the image mask is an irregular shape, which is not convenient for subsequent processing. Second, each pixel in regional saliency detection has its corresponding confidence, which is represented by its grayscale value. The closer the value is to 255, the higher the confidence, and vice versa, the closer the value is to 0, the lower the confidence. Usually, the edge processing confidence is low, resulting in unclear boundaries. The final output must be a definite slice of the original image, and the fuzzy boundaries cannot be retained. In the bounding box obtained according to regional saliency detection, it is necessary to retain the true positive area as much as possible, thereby preserving the integrity of the foreground area information, and to try to completely suppress the false positive area, so as to minimize the complexity of subsequent processing.
[0090] According to the above analysis, in this stage, a threshold needs to be determined to remove the fuzzy boundary, and then a bounding box is obtained based on the determined boundary. Therefore, the stage is divided into two steps. The first step is to adaptively select a threshold, retain pixels above the threshold, and remove pixels below the threshold to obtain a clear segmentation boundary. The second step is to obtain multiple accurate vertical bounding boxes based on the retained pixels as the second region of interest.
[0091] In the adaptive threshold selection step, the threshold is mainly determined by statistically analyzing the pixel distribution and pixel value characteristics of the saliency detection feature map, so as to distinguish the foreground and background with the largest possible inter-class variance. First, the grayscale values of all non-zero pixels are counted and the grayscale value distribution curve is drawn. According to the curve and the peak value of the curve, if the curve is unimodal, based on statistical theory, the normal distribution curve is fitted, and the vertex origin side is selected as the adaptive threshold. The pixels of the saliency detection are binarized according to the threshold. If it is greater than the threshold, it is considered to be the foreground area, otherwise it is considered to be the background.
[0092] If the curve is bimodal, that is, the foreground and background each have their own peaks, the peak valley is used as the segmentation threshold. Of course, this step requires setting the minimum threshold hyperparameter to adapt to the negative image. After experimental adjustment, the median of the gray value is selected as the minimum credible threshold. When the processing requirements are not very stringent, the maximum inter-class difference method can be used to exhaustively search for the segmentation threshold with the lowest probability of misclassification. According to the deformed inter-class variance calculation formula, the threshold with the maximum variance is taken to better distinguish the foreground and background.
[0093] In the subsequent steps, all foreground pixels are first traversed, and all foreground pixels connected to the first foreground pixel obtained by the traversal are searched in eight directions. This is regarded as a connector. During the search, the x, y coordinates of the upper left corner and the lower right corner of the connector are updated simultaneously. Finally, N vectors of length four are output as the bounding box results of the regional saliency detection screening.
[0094] According to this step, one or more second regions of interest may be obtained according to the regular segmentation boundaries, and the obtained second regions of interest may include an airport area, a port area, a parking lot area, and the like.
[0095] S103: The first region of interest and the second region of interest whose similarity is greater than a preset threshold are merged to obtain a merged region of interest.
[0096] Regional saliency detection can detect non-offset regional bounding boxes at various resolutions and in some special cases. However, when the target area is small or the low-level features are difficult to distinguish from the background, it is usually not effectively detected. In addition, the confidence of saliency detection has a large deviation. The advantage of target detection is that as long as the region contains specific targets, the approximate location of the region can be detected, and the confidence calculated based on each target is more realistic, but correspondingly, the bounding box has a large offset relative to the true value, and when there is no specific target in the region, the algorithm completely fails. Therefore, in the fusion selection of the two algorithms, the advantages of each algorithm should be retained and the disadvantages of the algorithm should be removed.
[0097] In a possible implementation, the intersection-and-union ratio of the first region of interest and the second region of interest is calculated, and the similarity is obtained according to the intersection-and-union ratio; the first region of interest and the second region of interest whose similarity is greater than a preset threshold are fused, and the bounding box of the second region of interest is used as the bounding box of the fused region of interest. The value of the preset threshold can be set according to actual conditions.
[0098] Specifically, in regional fusion, whether two regions can be fused is mainly determined by their similarity. Since the detection results of saliency detection and target detection are only related to the regional position and regional confidence, and the overlap of the regional positions accounts for the absolute majority of the factors affecting the relationship between the two, the similarity judgment condition is simplified, and the intersection-over-union ratio between regions is used as the similarity. For example, the ratio of the intersection of the target detection result and the saliency detection result to the smaller bounding box of the two is calculated as the intersection-over-union ratio. If the intersection-over-union ratio is high, it is considered that the region can be fused.
[0099] Therefore, if the first ROI has a high similarity with the second ROI, the unbiased bounding box of the second ROI detected by saliency is used as the fusion baseline, and the corresponding bounding box is deleted from the detection sequences of the two, and the bounding box vector of the second ROI detected by saliency is added to the final candidate result sequence.
[0100] For other unfused first ROIs and second ROIs with lower similarity, as long as the saliency detection or target detection result sequence is not empty, they are added to the final candidate result sequence, but their corresponding confidence is much lower than the result sequence of the fusion of the two, so that the confidence information can be retained in this step.
[0101] S104: ranking the fused ROI and the unfused first ROI and second ROI in terms of relative importance, and obtaining the selected ROI according to the ranking result.
[0102] Region sorting mainly considers two factors: the importance of the region and the confidence of the region. The importance of different types of regions can be configured by modifying the configuration file, and the importance of the same type of regions is determined by the distribution of the objects of interest. The confidence of the bounding box after the fusion of regional saliency detection and object detection will be different, which mainly depends on the confidence threshold of the saliency detection, the number of credible pixels and the confidence of each object in the object detection detection box.
[0103] First, according to the target category in the region of interest to be screened, the fused region of interest and the unfused first region of interest and the second region of interest containing the same target category are obtained. For example, if the region of interest to be screened is an airport area, the fused region of interest and the unfused first region of interest and the second region of interest belonging to the airport category (the target in the region is an airplane) are obtained, and the region of interest of this type is of high importance.
[0104] Furthermore, the confidence of the regions of interest containing the same target category is calculated.
[0105] In this stage, all relevant influencing factors are first normalized to the range of 0 to 1 to ensure that the magnitude of each parameter is consistent, which is convenient for adjusting parameters in the experiment. The confidence threshold in significance detection is denoted as T s The lower the value, the lower the overall credibility of the detected area. The foreground pixel of the region in saliency detection is denoted as PN roi The more the number, the higher the confidence of the area. When normalizing, the number of pixel values of the detected image is taken as 1, recorded as PN ori , the ratio of regional pixels to detected image pixels is taken as the normalization result.
[0106] The confidence of the target detection bounding box cannot simply take the product of the confidence of a single target as the confidence of its bounding box, which will result in a higher confidence for the bounding box containing fewer targets, which is contrary to reality. The confidence of the bounding box result of target detection mainly depends on three factors. The first is the product of the confidence of all targets of interest. If the confidence of a single target is recorded as score, the higher the value, the higher the overall confidence. The second is the density of the targets of interest. If there are more targets under the same bounding box area, or the number of targets is the same but the distribution is denser, both of which can be considered that the area is more important. This metric takes into account both the number of targets of interest and the area of the targets of interest in pixels. Therefore, for each slice, the confidence is calculated as follows, and the final result is normalized by the logarithmic normalization function.
[0107]
[0108] Among them, W1, W2, W3, and W4 represent weight parameters, PN roi Represents the foreground pixel in the region of saliency detection, PN ori represents the pixel of the detected image, T s Represents the confidence threshold in saliency detection, score i represents the confidence of the i-th target in target detection, S roi Indicates the number of targets in the bounding box in target detection, S ori Represents the area of the bounding box in target detection. After many experiments, W1, W2, W3, and W4 were finally determined to be 2.3, 1.2, 4.1, and 1.4 respectively.
[0109] The regions of interest containing the same target category are sorted from large to small according to the confidence level, and a preset number of regions of interest at the front are used as the screened regions of interest.
[0110] In an optional implementation, the regions of interest classified as airport areas are sorted from large to small according to the confidence, and the region of interest ranked first is output to obtain the selected regions of interest. The regions of interest classified as port areas can also be sorted from large to small according to the confidence, and the region of interest ranked first is output to obtain the selected regions of interest.
[0111] In an exemplary scenario, in a 20,000-pixel-side, three-channel remote sensing image, there are up to 12*108 pixels. If each pixel is stored in uint8 and other information that needs to be stored in the image is not considered, the image size is about 1.2GB. Assuming that the average uplink and downlink speeds in the laboratory are 5MB / s, the average upload and download time is about 4 minutes. In addition, when such an image is opened with lightweight software, the response time is about 5 minutes. When viewing the content of the image, it will not be able to respond quickly or run normally to other applications because the image is too large and occupies the memory. In civil use, it is often necessary to accurately and quickly locate important locations such as airports and ports through remote sensing images, but because the number and types of instances contained in them are quite numerous, it is very necessary to quickly, accurately and precisely filter out the areas of interest from these pixels.
[0112] To this end, the present application designs a regional intelligent screening system. The system integrates four key technologies: regional intelligent screening based on target detection, regional intelligent screening based on regional significance detection, regional fusion and regional importance sorting. In this system, for the input large-format remote sensing image, the target detection algorithm is first used to locate the target of interest, such as an airplane or a ship. Secondly, the regional significance detection algorithm is used to locate the significant areas in the large-format remote sensing image, such as airports or ports. Finally, based on the results of target detection and significance detection, the regions of interest are further generated, sorted according to importance, and the screening results are output.
[0113] According to a method for screening regions of interest in remote sensing images provided in an embodiment of the present application, regions of interest can be accurately and quickly extracted from large-format remote sensing images and transmitted, achieving 30-50 times compression without losing important information, greatly reducing bandwidth requirements.
[0114] The present application also provides a device for screening regions of interest in remote sensing images. The device is used to perform the method for screening regions of interest in remote sensing images in the above-mentioned embodiment. Figure 8 As shown, the device comprises:
[0115] The first detection module 801 is used to detect the target of interest in the remote sensing image according to the target detection algorithm, and generate a target bounding box, and obtain one or more first regions of interest according to the target bounding box;
[0116] A second detection module 802 is used to detect a salient region in the remote sensing image according to a regional saliency detection algorithm to obtain one or more second regions of interest;
[0117] A fusion module 803 is used to fuse the first region of interest and the second region of interest whose similarity is greater than a preset threshold to obtain a fused region of interest;
[0118] The sorting and screening module 804 is used to sort the relative importance of the fused regions of interest and the unfused first and second regions of interest, and obtain the screened regions of interest according to the sorting results. It should be noted that the device for screening regions of interest in remote sensing images provided in the above embodiment only uses the division of the above functional modules as an example when executing the method for screening regions of interest in remote sensing images. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device for screening regions of interest in remote sensing images provided in the above embodiment and the method for screening regions of interest in remote sensing images are of the same conception, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.
[0119] The embodiment of the present application also provides an electronic device corresponding to the method for screening regions of interest in remote sensing images provided in the above-mentioned embodiment, so as to execute the above-mentioned method for screening regions of interest in remote sensing images.
[0120] Please refer to Fig. 9 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Fig. 9 As shown, the electronic device includes: a processor 900, a memory 901, a bus 902 and a communication interface 903, and the processor 900, the communication interface 903 and the memory 901 are connected via the bus 902; the memory 901 stores a computer program that can be run on the processor 900, and when the processor 900 runs the computer program, it executes the method for screening the area of interest in the remote sensing image provided by any of the aforementioned embodiments of the present application.
[0121] The memory 901 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 903 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0122] The bus 902 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 901 is used to store programs, and the processor 900 executes the programs after receiving the execution instructions. The method for screening the region of interest in the remote sensing image disclosed in any implementation of the embodiment of the present application may be applied to the processor 900, or implemented by the processor 900.
[0123] The processor 900 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 900. The above processor 900 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 901, and the processor 900 reads the information in the memory 901 and completes the steps of the above method in combination with its hardware.
[0124] The electronic device provided in the embodiment of the present application and the method for screening regions of interest in remote sensing images provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.
[0125] The present application also provides a computer-readable storage medium corresponding to the method for screening regions of interest in remote sensing images provided in the above embodiments. Fig.10 The computer-readable storage medium shown is a CD 1000 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the method for screening regions of interest in remote sensing images provided in any of the aforementioned embodiments will be executed.
[0126] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.
[0127] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method for screening regions of interest in remote sensing images provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0129] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.
Claims
1. A method for screening regions of interest in remote sensing images, characterized in that: include: Detecting a target of interest in the remote sensing image according to a target detection algorithm, and generating a target bounding box, and obtaining one or more first regions of interest according to the target bounding box; Detecting a salient region in the remote sensing image according to a regional saliency detection algorithm to obtain one or more second regions of interest; Merging the first region of interest and the second region of interest whose similarity is greater than a preset threshold to obtain a fused region of interest; The fused region of interest and the unfused first region of interest and the second region of interest are ranked in relative importance, and the selected region of interest is obtained according to the ranking result.
2. The method according to claim 1, characterized in that: Obtaining one or more first regions of interest according to the target bounding box includes: Enlarging the target bounding box according to a preset ratio to obtain an enlarged target bounding box; Determine whether the enlarged target bounding boxes intersect, and merge the intersecting enlarged target bounding boxes; Determine whether the side length of the enlarged target enclosing frame is greater than a preset threshold, and determine whether the distance between two enlarged target enclosing frames whose side lengths are greater than the preset threshold is less than the preset threshold; If the side length of the enlarged target bounding box is greater than a preset threshold, and the distance between the two enlarged target bounding boxes whose side lengths are greater than the preset threshold is less than the preset threshold, the two enlarged target bounding boxes are merged; One or more first regions of interest are generated according to the merged target bounding boxes.
3. The method according to claim 1, characterized in that Detecting a salient region in the remote sensing image according to a regional saliency detection algorithm to obtain one or more second regions of interest includes: Inputting the remote sensing image into a regional saliency detection model to obtain one or more detected irregular saliency maps; The pixels in the irregular saliency map that are greater than or equal to the segmentation threshold are retained, and the pixels that are less than the segmentation threshold are removed to obtain a regular segmentation boundary; One or more second regions of interest are obtained according to the regular segmentation boundaries.
4. The method according to claim 3, characterized in that: Inputting the remote sensing image into a regional saliency detection model to obtain one or more detected irregular saliency maps, including: Inputting the remote sensing image into the regional saliency detection model, reducing the remote sensing image by bilinear interpolation and random cropping to obtain a reduced remote sensing image; Inputting the reduced remote sensing image into a fully convolutional neural network to obtain a predicted rough saliency map; The rough saliency map and the remote sensing image to be detected are input into a residual refinement module to obtain one or more detected irregular saliency maps.
5. The method according to claim 1, characterized in that The first region of interest and the second region of interest whose similarity is greater than a preset threshold are merged to obtain a merged region of interest, including: Calculate the intersection-and-union ratio of the first region of interest and the second region of interest, and obtain the similarity according to the intersection-and-union ratio; The first region of interest and the second region of interest whose similarity is greater than a preset threshold are fused, and the bounding box of the second region of interest is used as the bounding box of the fused region of interest.
6. The method according to claim 1, characterized in that The fused ROI and the unfused first ROI and second ROI are ranked in relative importance, and the selected ROIs are obtained according to the ranking results, including: According to the target category in the region of interest to be screened, obtaining the fused region of interest and the unfused first region of interest and the region of interest containing the same target category in the second region of interest; Calculate the confidence of the region of interest containing the same target category; The regions of interest containing the same target category are sorted from large to small according to the confidence level, and a preset number of regions of interest at the front are used as the screened regions of interest.
7. The method according to claim 6, characterized in that Calculate the confidence of the region of interest containing the same target category, including: The confidence level is calculated according to the following formula: Among them, W1, W2, W3, and W4 represent weight parameters, PN roi Represents the foreground pixel in the region of saliency detection, PN ori represents the pixel of the detected image, T s Represents the confidence threshold in saliency detection, score i represents the confidence of the i-th target in target detection, S roi Indicates the number of targets in the bounding box in target detection, S ori Represents the area of the bounding box in target detection.
8. A device for screening regions of interest in remote sensing images, characterized in that: include: A first detection module, configured to detect a target of interest in a remote sensing image according to a target detection algorithm, and generate a target bounding box, and obtain one or more first regions of interest according to the target bounding box; A second detection module, configured to detect a salient region in the remote sensing image according to a regional saliency detection algorithm, and obtain one or more second regions of interest; A fusion module, used for fusing the first region of interest and the second region of interest whose similarity is greater than a preset threshold to obtain a fused region of interest; The sorting and screening module is used to sort the relative importance of the fused regions of interest and the unfused first regions of interest and second regions of interest, and obtain the screened regions of interest according to the sorting results.
9. An electronic device, characterized in that: The invention comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the method for screening regions of interest in remote sensing images according to any one of claims 1 to 7 when executing the program instructions.
10. A computer-readable medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions are executed by a processor to implement a method for screening a region of interest in a remote sensing image as described in any one of claims 1 to 7.
