A remote sensing image detection and processing method and system based on density mask
Through the remote sensing image detection processing method based on density mask, the accuracy problem of the remote sensing object detector when detecting the multi-directional attitude rotation characteristic target is solved, and more accurate acquisition and identification of target object position information is achieved, reducing the calculation amount. Improved identification robustness.
Patent Information
- Application Number
- CN202410518779.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-04-28
AI Technical Summary
When existing remote sensing object detectors detect targets with multi-directional attitude rotation characteristics, it is difficult to perform compact and precise positioning. Due to the different contributions to the rotation feature extraction, the target detection is inaccurate, and it is impossible to dynamically determine the inclination angle and number of objects. Causes missed inspection or false alarm.
Using the remote sensing image detection processing method based on density mask, the network model is generated through the density map to generate the density map, and the pixel level inversion operation is performed to obtain the density mask image, determine the number of evaluations of the object tilt angle and the weight sequence, and input the feature extraction network Perform weighted combined feature map extraction to identify and locate target objects.
It realizes more accurate acquisition of target object position information, dynamically predicts object tilt angle, reduces detection calculations, improves recognition robustness and accuracy, and is especially suitable for feature equalization extraction of object tilt angle or rotation state in complex environments and Detection and identification.
Smart Images

Figure CN118279746B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a remote sensing image detection and processing method and system based on density mask. Background Art
[0002] Most of the existing mainstream object detectors are based on convolutional neural networks (CNNs), which can be roughly divided into two categories: two-stage detectors and one-stage detectors. Mainstream detectors first extract image features through neural network convolution operations, preset a series of prior horizontal anchor boxes on the feature map, and then perform classification and regression based on these anchor boxes and feature maps.
[0003] Since the existing convolutional structure lacks the ability to extract rotation-invariant features, and the horizontal frame positioning method cannot accurately describe the attitude information of remote sensing targets, the existing remote sensing target detectors are difficult to perform compact and accurate positioning when detecting targets with multi-directional attitude rotation characteristics. To address the above problems, the current mainstream remote sensing target detectors have been improved in two directions: enhanced target rotation feature extraction and flexible and efficient target representation.
[0004] In the remote sensing images actually taken, different types of objects covered by an image are usually densely distributed in different areas at different angles. The area corresponding to each angle is different, and the contribution of different areas to rotation feature extraction is generally different. Therefore, there are problems such as inaccurate target detection caused by different contribution levels of rotation feature extraction.
[0005] The existing mainstream detectors do not take into account the fact that the objects covered in the actual remote sensing images may appear at a limited number of angles, and still perform invalid detections at many unnecessary angles during the target detection task. Recently, although a prediction method for the possible rotation angles in a remote sensing image has been proposed, the number of rotation angles of the objects covered in an image is generally selected based on a predetermined value before prediction. Due to the lack of prior information on the number of angles, there will be a difference between the predetermined number of angles and the actual number of angles. In terms of the quality of the target detection task, the difference in the number of angles will lead to problems such as missed detection or false alarms of objects in the image. At the same time, there is also the problem of a large amount of calculation caused by too many angles.
[0006] Therefore, it is necessary to provide a remote sensing image detection and processing method based on density mask to solve the above problems. Summary of the invention
[0007] The present invention aims to provide a remote sensing image detection and processing method and system based on density mask to solve the technical problems in the prior art that the inclination angles and the number of objects in the remote sensing image cannot be dynamically determined, the number of inclination angles of objects in the remote sensing image cannot be dynamically determined, resulting in missed detection or false alarm of objects in the image, a large amount of calculation due to too many angles, and inaccurate target detection due to different contribution levels of rotation feature extraction. The technical problems to be solved by the present invention are achieved through the following technical solutions.
[0008] The first aspect of the present invention proposes a remote sensing image detection and processing method based on density mask, comprising: based on different application target detection tasks, obtaining a remote sensing image to be processed that is taken in real time, and the application target detection tasks include air traffic management detection tasks and ship supervision detection tasks; using a density map generation network model to generate a density map corresponding to the remote sensing image to be processed; performing a pixel-level inversion operation on the basis of the generated density map to obtain a density mask image corresponding to the remote sensing image to be processed, determining an evaluation number of the object's inclination angle based on the density mask image, and further predicting an object inclination angle sequence and a weight sequence; inputting the remote sensing image to be processed, the obtained object inclination angle sequence and the weight sequence into a feature extraction network model to obtain a weighted combination feature map corresponding to the remote sensing image to be processed in different application target detection tasks; according to the obtained weighted combination feature map, performing target object recognition and positioning that matches the different application target detection tasks to obtain recognition and positioning information.
[0009] According to an optional implementation manner, determining the evaluation number of the inclination angle of the object based on the density mask image includes: determining the number of pixels in the density mask image whose pixels are specific pixel values, and calculating the area ratio of the number of pixels in the density mask image whose pixels are specific pixel values to the total number of pixels using the following expression:
[0010] ;
[0011] in, Indicates the area ratio of the number of pixels having a specific pixel value to the total number of pixels in the density mask image; Indicates the number of pixels in the density mask image whose pixels are specific pixel values; Represents the total number of pixels of all pixels in the density mask image; and determines the evaluation number of the tilt angle of the object in the density mask image according to the calculated area ratio.
[0012] According to an optional implementation, it further includes: through a deep convolution layer, using a number of convolution kernels equal to the evaluated number of object inclination angles, performing an affine rotation transformation according to the obtained object inclination angle sequence, and obtaining a transformed and updated convolution kernel; according to the transformed and updated convolution kernel, performing average pooling on the remote sensing image to be processed after a deep convolution operation, and then passing through a combination layer of linear weighting and nonlinear functions to obtain an object inclination angle sequence and a weight sequence, wherein the weight sequence refers to a sequence formed by weight values corresponding to each object inclination angle.
[0013] According to an optional implementation, the use of a density map generation network model to generate a density map corresponding to the remote sensing image to be processed includes: inputting the remote sensing image to be processed into the density map generation network model, specifically passing through a multi-layer CNN convolutional network and linearly adding to obtain a generation network result, that is, a density map corresponding to the remote sensing image to be processed, wherein the multi-layer CNN convolutional network includes a three-layer CNN convolutional network.
[0014] According to an optional implementation, a pixel-level inversion operation is performed on the basis of the generated density map to obtain a density mask image corresponding to the remote sensing image to be processed, including: performing a sliding traversal operation on the generated density map in accordance with a specified step size by means of a sliding window, and dynamically adjusting the size and number of hot zones of captured objects to obtain a density mask image corresponding to the remote sensing image to be processed.
[0015] According to an optional implementation, a sliding traversal operation is performed by adjusting the sliding window according to different specified step sizes, and a sliding traversal operation is performed, that is, a sampling operation is performed; in each sampling operation, a summation operation is performed on all pixel values in the current window to obtain a summation value, and the calculated summation value is compared with a predetermined threshold to determine the size and number of captured object hot spots; the length of the sliding window is in the range of 20 pixels to 50 pixels, and the width of the sliding window is in the range of 20 pixels to 50 pixels; the specified step size is in the range of 5 pixels to 10 pixels.
[0016] According to an optional implementation, the remote sensing image to be processed, the obtained object tilt angle sequence and the weight sequence are input into the feature extraction network model to obtain a weighted combined feature map corresponding to the remote sensing image to be processed in different application target detection tasks, including: using a number of convolution kernels equal to the number of object tilt angle evaluations, performing a convolution operation on the feature map of the input deep convolution layer, and performing weighted summation on the features extracted by each deep convolution layer according to the weight sequence:
[0017] ;
[0018] Where y represents the weighted combined feature map output by the deep convolutional layer, and y is a matrix containing the following information: rotation angle, object position, and shape; represents the convolution kernel after transformation and update, n represents the number of convolution kernels, and n is a positive integer; Represents the weight corresponding to the object tilt angle sequence, which is used to transform and update the convolution kernel. n is a positive integer; Represents a convolution operation; x represents the feature map of the input depth convolution layer.
[0019] According to an optional implementation manner, the obtained weighted combined feature map is input into a detection and recognition model to obtain recognition and positioning information and target object recognition results corresponding to the remote sensing image to be processed, wherein the detection and recognition model is a single-stage target detection model; the recognition and positioning information includes the position information of the aircraft in the parking area relative to the reference axis, the position information of the wing, the position information of the fuselage, the position information of the runway, and the position information of the ship in the specific area relative to the reference axis; the target object recognition results include: whether aircraft and ships are included, the type of aircraft, the type of ship, the number of target objects, and the object inclination angle of each target object relative to the reference axis.
[0020] The second aspect of the present invention provides a remote sensing image processing system based on density mask, which is used to execute the remote sensing image detection and processing method based on density mask described in the first aspect of the present invention, and the remote sensing image processing system includes: an acquisition processing module, which acquires the remote sensing image to be processed taken in real time based on different application target detection tasks, and the application target detection tasks include air traffic management detection tasks and ship supervision detection tasks; a generation processing module, which uses a density map generation network model to generate a density map corresponding to the remote sensing image to be processed; a prediction processing module, which performs a pixel level inversion operation on the basis of the generated density map to obtain a density mask image corresponding to the remote sensing image to be processed, determines the evaluation number of the object inclination angle based on the density mask image, and further predicts the object inclination angle sequence and weight sequence; an extraction processing module, which inputs the remote sensing image to be processed, the obtained object inclination angle sequence and the weight sequence into the feature extraction network model, and obtains a weighted combination feature map corresponding to the remote sensing image to be processed in different application target detection tasks; an identification and determination module, which performs target object identification and positioning matching different application target detection tasks according to the obtained weighted combination feature map, and obtains identification and positioning information.
[0021] The embodiments of the present invention include the following advantages:
[0022] Compared with the prior art, the present invention generates a density map corresponding to the remote sensing image to be processed, and generates a density mask image by performing a pixel-level inversion operation on the basis of the density map; determines the evaluation number of the object's tilt angle based on the density mask image, and further predicts the object's tilt angle sequence and weight sequence, and performs feature extraction based on the obtained object tilt angle sequence and weight sequence and the remote sensing image to be processed as the model input of the feature extraction network model, so as to obtain a weighted combination feature map after feature enhancement of the target object, and obtain a weighted combination feature map containing more effective target object features, and then uses the obtained weighted combination feature map for detection and recognition, so as to obtain more accurate target object location information, dynamically predict the tilt angle of the target object in the remote sensing image to be processed, and thus accurately identify target objects with different object tilt angles, and also reduce the amount of calculation in the detection and recognition calculation process.
[0023] In addition, the present invention is particularly suitable for feature balanced extraction and detection and recognition of one or more target objects at an inclined angle or a rotational state in a complex environment, and can effectively improve the robustness and accuracy of recognition and detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flow chart of an example of a remote sensing image detection and processing method based on density mask of the present invention;
[0025] Figure 2 It is a principle schematic diagram of the remote sensing image detection and processing method based on density mask of the present invention;
[0026] Figure 3 It is a schematic diagram of the principle of generating a density mask image in the remote sensing image detection and processing method based on density mask of the present invention;
[0027] Figure 4 It is a schematic diagram of the principle of generating an object tilt angle sequence and a weight sequence in the remote sensing image detection processing method based on density mask of the present invention;
[0028] Figure 5 It is a schematic diagram of the principle of the remote sensing image detection and processing method based on density mask of the present invention from another angle;
[0029] Figure 6 is a schematic diagram of an example of a remote sensing image detection and processing system based on a weighted combined feature map of the present invention;
[0030] Figure 7 is a schematic structural diagram of an electronic device embodiment according to the present invention;
[0031] Figure 8 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION
[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. In the present invention, the upper surface of the object in the accompanying drawings is the upper surface, and the lower surface of the object in the accompanying drawings is the lower surface. This is only to more clearly illustrate the automatic recognition process of handwritten forms based on deep learning, and it cannot be understood as a limitation of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0033] In view of the above problems, the present invention provides a remote sensing image detection and processing method based on density mask. The method uses an adaptive balanced density map inversion network (Adaptive Balance Density-inversion Network, also abbreviated as "ABD-Net") to specifically first generate a density map corresponding to the remote sensing image to be processed, and then perform a pixel-level inversion operation on the basis of the density map to generate a density mask image; based on the density mask image, an evaluation quantity of the object's inclination angle is determined, and the object's inclination angle sequence and weight sequence are further predicted, and feature extraction is performed based on the obtained object inclination angle sequence and weight sequence and the remote sensing image to be processed as the model input of the feature extraction network model, so as to obtain a weighted combined feature map after feature enhancement of the target object, and then use the obtained weighted combined feature map for detection and identification, so as to obtain more accurate location information of the target object.
[0034] It should be noted that in the present invention, the adaptive balanced density map inversion network adds three modules to the existing detector (for example, the front part of the backbone network of the detection model), specifically including: a density map mask generation module (Density-Map and Mask Generation Module, i.e., DGM), a mask routing prediction module (Mask Routing Prediction Module, i.e., MRM) and a spatial balance calculation module (Spatial-Balance Calculation Module, i.e., SCM). For details, see Figure 2 and Figure 3 These three modules are used in conjunction with the feature extraction network (such as ResNet) of an existing detector or detection model (a new model independent of the feature extraction network of an existing detector or detection model), or these three modules and a self-built feature extraction network together form a new backbone network (i.e., backbone), etc. The above three modules are seamlessly connected and well coupled, giving the entire network greater flexibility to detect targets with different orientations.
[0035] In addition, ABD-Net also contains a Feature Pyramid Network, a Feature Alignment Module (FAM) and an Oriented Detection Module (ODM). FAM and ODM constitute the detection head applied to each scale of the feature pyramid.
[0036] Example 1
[0037] Reference Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 The first aspect of the present invention provides a remote sensing image detection and processing method based on density mask.
[0038] First, in step S101, based on different application target detection tasks, remote sensing images to be processed that are taken in real time are acquired, and the application target detection tasks include air traffic management detection tasks and ship supervision detection tasks.
[0039] In a specific implementation, the application target detection task is, for example, an air traffic management detection task. For example, an aerial remote sensing image taken by a satellite, a drone or other high-altitude remote sensing platform is obtained as the remote sensing image to be processed, and the remote sensing image to be processed covers a designated airport area, specifically including runways, taxiways, aprons, terminals, control towers, navigation facilities and other ancillary facilities.
[0040] For example, in a remote sensing image containing a large number of buildings, vegetation or other complex terrain, ships or aircraft in a specific area (such as aircraft at an airport) may present different angles of posture due to different parking positions and shooting angles, and need to be accurately located and identified; for aircraft, the importance of their wings and fuselage in determining the rotation angle may be different, and it is necessary to intelligently adjust the focus on these areas to ensure the accurate extraction of the overall rotation properties.
[0041] In a specific implementation, the application target detection task is, for example, a ship supervision detection task. The remote sensing image to be processed is a remote sensing image of a port area.
[0042] In another specific implementation, the application target detection task is, for example, an adversarial training detection task. The remote sensing image to be processed is an adversarial training area image.
[0043] It should be noted that the above is only provided as an optional example and is not to be construed as a limitation to the present invention.
[0044] Next, in step S102, a density map generation network model is used to generate a density map corresponding to the remote sensing image to be processed.
[0045] Specifically, a density map generation network model is constructed, which specifically corresponds to the density map mask generation module (DGM) of the adaptive balanced density map inversion network. MCNN (Multi-Column Convolutional Neural Network) is used as the backbone network (i.e., backbone) of the density map generation network model. In the density map generation network model, multi-column CNNs (in other words, multi-layer CNN networks) are used, and each column uses convolution kernels of different sizes to correspond to the center position (head location) of objects of different sizes in the two-dimensional image plane. In addition, two pooling layers are introduced into MCNN, so the output feature map of height and width will be reduced by 4 times. In order to retain the original resolution, sampling is performed on the density map generated by cubic interpolation and enlarged by 4 times to restore the original resolution. For cases where the image height or width is not a multiple of four, the remote sensing image to be processed can be directly adjusted to its original resolution.
[0046] For the training process of the density map generation network model, a remote sensing image with the category label information of the target object marked (the category label is discrete, and each category label represents one of the predefined category sets) is used to establish a first training data set, such as an aerial remote sensing image with the runway, taxiway, apron, terminal, tower, navigation facilities, aircraft location information, and the angle of the target object (such as aircraft, tower) relative to the horizontal axis (i.e., reference axis) in the specified direction (also referred to as "object tilt angle"), and multiple category labels and object tilt angle labels are used to define positive samples and negative samples. Then, the density map generation network model is trained using the first training data set.
[0047] It should be noted that, using the target bounding box, a rectangle or rectangular box surrounding each target object is drawn on the remote sensing image to be processed, and this target bounding box accurately determines the specific position of the target object in the remote sensing image to be processed. The target bounding box is defined by the pixel coordinates (x1, y1, x2, y2) of its upper left corner and lower right corner, or described by the center point coordinates (xc, yc) and the width (w) and height (h). In addition, for multi-category labels, a category label is assigned to each labeled target object, indicating the object category to which it belongs. The category label is discrete, and each category label represents an item in a predefined category set. In addition, key point annotation is also included. For tasks that require accurate capture of the internal structure of the target object or the location of a specific part, key points on the target object will also be marked, such as significant geometric features of a specific object, areas of a specific shape, etc. Key points are marked, for example, in the form of coordinate pairs or multi-point coordinates.
[0048] For the calibration of positive and negative samples, the intersection of the candidate area to be selected and the real bounding box is used to determine whether it is a positive sample. For example, a specific threshold (such as 0.5) is set. If the IoU between the candidate area to be selected and a real bounding box exceeds this specific threshold, the candidate area to be selected is considered to be a positive sample for the real target. If it is less than the specific threshold, the candidate area to be selected is a negative sample for the real target. This means that when the candidate area to be selected overlaps with the real target to a considerable extent, the density map generation network model should recognize the image sample (i.e., a remote sensing image to be processed) as containing the existence of a target object of a certain category.
[0049] It should be noted that the network algorithm of the invention also includes preprocessing operations on the input remote sensing image to be processed, including image scaling and data enhancement.
[0050] Gradient descent back propagation is adopted to update the network parameters using the generation result (i.e., density map) of the density map generation network model and the true value in the first training data set (specifically, the remote sensing image with labeled information). The network parameters refer to the weight parameters of various functional layers (convolutional layer, pooling layer, fully connected layer, etc.) in the density generation network model.
[0051] The loss function of the density map generation network model during training is calculated based on the pixel-level mean absolute error using the following expression.
[0052] ;
[0053] Where L(θ) represents the loss function of the density map generation network model during training based on the pixel-level mean absolute error; θ is the network weight parameter of the density map generation network model; N is the total number of remote sensing images to be processed in the first training data set, and N is a positive integer; X i Represents the input remote sensing image to be processed; D i It is i True density map; D(X i ; θ) is the density map generated by the density map generation network model.
[0054] Specifically, when the pixel-level mean absolute error of the density map generation network model shows a trend of convergence and when the number of training rounds is between 100 and 200, the training process of the density map generation network model is stopped. By optimizing the training process of the density map generation network model, the efficiency and accuracy of the generated density map can be improved.
[0055] In the first training data set, the adaptive kernel method is used to generate the real density map. All the remote sensing images to be processed in the first training data set are mapped one by one to generate the density map. Specifically, the Gaussian kernel is used to blur the target objects (such as aircraft, towers, ships, etc.) to generate the real density map. The diffusion parameter Calculated by the KNN (K-Nearest-Neighbor) method, the following formula:
[0056] , ,
[0057] in, F ( x ) represents the output graph output by the density map generation network model, i.e., the density map; x i It is i target objects (such as aircraft, towers, ships, etc.), i is a positive integer; N is the total number of remote sensing images to be processed in the first training data set, N is a positive integer, is the Gaussian kernel, which is Perform convolution operation, δ represents the Dirac function; represents the diffusion parameter; β represents the correlation coefficient, which is obtained according to the experimental results and is in the range of 0.1 to 0.5. For the air traffic management detection task, when the correlation coefficient is 0.3, the accuracy of density map generation is greater than the predetermined value (for example, 95%) and the effect is the best (it can clearly highlight the outer contour or designated part or designated area of the target object to be obtained); is the average distance of the K nearest target objects, and K represents the number of objects that are close to the target object.
[0058] Optionally, the correlation coefficient β is in the range of 0.1 to 0.5, which can clearly highlight the key objects (i.e., target objects, such as aircraft, aircraft wings or fuselage, ships, etc.) marked. For the application scenario of ship supervision and inspection tasks, the correlation coefficient β is in the range of 0.3 to 0.5, which can clearly highlight the key objects (i.e., target objects, such as ships, etc.) marked.
[0059] It should be noted that the remote sensing image to be processed for generating the density map is, for example, a remote sensing image with the positions of key objects (i.e., target objects) manually annotated. F ( x ), and combined with the manually labeled positions, generate a density map. The purpose of density map generation is to generate annotation labels, that is, key point annotations. Under the premise of manual annotation, the key points are processed to generate density maps, which can be more quantified.
[0060] The remote sensing image to be processed is input into the density map generation network model, and specifically passes through a multi-layer CNN convolutional network and undergoes linear addition to obtain a generation network result, i.e., a density map corresponding to the remote sensing image to be processed. For example, an aerial remote sensing image, specifically an image format of [C in , H, W] aerial remote sensing images are input into the trained density map generation network model, and the obtained feature maps (such as feature maps containing detailed features of the aircraft, local structural features, distribution structural features of the aircraft relative to the tower, or spatial structural features, such as the shape of the aircraft, the tilt angle of the aircraft relative to the reference axis, i.e., the rotation angle or orientation) are linearly added to obtain the density map corresponding to the remote sensing image to be processed. Among them, C in It represents the number of channels of the input remote sensing image to be processed, H represents the height of the input remote sensing image to be processed, and W represents the width of the input remote sensing image to be processed.
[0061] Specifically, for air traffic management detection tasks, through the convolution layer with a small convolution kernel (such as 3 pixels x 3 pixels or smaller), it is possible to capture the detailed features and local structural features of the remote sensing image to be processed, such as the shape of the aircraft, the object tilt angle of the aircraft relative to the reference axis, that is, the rotation angle or orientation, the wing features or fuselage features of the aircraft, the object tilt angle of the wing of the aircraft relative to the reference axis, the object tilt angle of the fuselage of the aircraft relative to the reference axis, etc. Through the convolution layer with a large convolution kernel (such as 5 pixels x 5 pixels, 7 pixels x 7 pixels or larger), it is possible to capture a wider range of spatial up-down structural information and global layout information, such as the distribution structure characteristics of the aircraft relative to the tower or the feature map of the spatial structure characteristics, the background information of the aircraft, and the location information of key objects such as the aircraft tower.
[0062] Specifically, the multi-layer CNN convolutional network includes a three-layer CNN convolutional network.
[0063] It should be noted that, in this example, the remote sensing image to be processed is input into the density map generation network model (corresponding to the density map mask generation module). The density generation network model includes three columns of CNN, that is, a three-layer CNN convolutional network. The first column undergoes 5*5 (i.e., 5 pixels * 5 pixels) convolution, two 3*3 (i.e., 3 pixels x 3 pixels) convolutions + 2*2 (2 pixels x 2 pixels) pooling, and 3*3 (3 pixels x 3 pixels) convolutions; the second column undergoes 7*7 (7 pixels x 7 pixels) convolutions, two 5*5 (5 pixels x 5 pixels) convolutions + 2*2 (2 pixels x 2 pixels) pooling, and 5*5 (5 pixels x 5 pixels) convolutions; the third column undergoes 9*9 (9 pixels x 9 pixels) convolutions, two 7*7 (7 pixels x 7 pixels) convolutions + 2*2 (2 pixels x 2 pixels) pooling, and 7*7 (7 pixels x 7 pixels) convolutions. For details, please refer to Figure 3 . The feature maps obtained by the three-column CNN are linearly added to obtain the result of the generation network, that is, the density map corresponding to the remote sensing image to be processed. Each column of the multi-column CNN will obtain a feature map through convolution pooling operation, and the feature map of each column is added to obtain a total feature map (Note: the total feature map here is the total feature map in the density map generation network). In other application detection tasks, different remote sensing images to be processed also correspond to different density maps. The above is only explained as an optional example and cannot be understood as a limitation of the present invention.
[0064] Next, in step S103, a pixel-level inversion operation is performed on the basis of the generated density map to obtain a density mask image corresponding to the remote sensing image to be processed, and the evaluation quantity of the object's tilt angle is determined based on the density mask image, and the object's tilt angle sequence and weight sequence are further predicted.
[0065] Specifically, the generated density map is traversed by sliding the window in accordance with the specified step size, and the size and number of the hot zones of the captured objects are dynamically adjusted to obtain a density mask image corresponding to the remote sensing image to be processed.
[0066] By adjusting the sliding window to perform a sliding traversal operation according to different specified step sizes, a sliding traversal operation is performed, that is, a sampling operation is performed. In each sampling operation, all pixel values in the current window are summed to obtain a sum value, and the calculated sum value is compared with a predetermined threshold to determine the size and number of the captured object hot zone.
[0067] Optionally, the length of the sliding window is in the range of 20 pixels to 50 pixels, the width of the sliding window is in the range of 20 pixels to 50 pixels, and the specified step length is in the range of 5 pixels to 10 pixels.
[0068] In a specific implementation, the density-map and mask generation module (DGM) sets corresponding predetermined thresholds for different application target detection tasks, for example, from 0.001 to 1, with a step length of 10 times. Specifically, 0.001, 0.01, 0.1, so that the density mask image can contain the target object, such as a designated area of an aircraft (such as a wing, fuselage, etc.), a bow or stern of a ship, etc.
[0069] The corresponding predetermined threshold is set for different application target detection tasks. The smaller the predetermined threshold, the larger the area of the density map mask and the smaller the discrimination, for example, ships and ports will be covered together. The larger the predetermined threshold, the smaller the area of the density map mask and the higher the discrimination. In practical applications, in order to compromise, the threshold is generally taken as 0.01, which can effectively distinguish objects with obvious boundaries, such as airplanes and towers in airports.
[0070] The density map mask generation module performs pixel-level inversion operations on the basis of the density map to generate a mask, filter out pixel values in target-free or limited target areas, and better highlight regional features (regional features including target objects).
[0071] Specifically, the density map generated by the density map generation network model is used as input, and the sliding window size W is empirically determined, for example, the length is 20 to 50 pixels, and the width is 20 pixels. The sliding window is traversed from top to bottom and from left to right in the entire image of the density map (also called sliding value sampling), and the specified step size is 5 pixels to 10 pixels. In each sampling, all pixel values in the current window are summed, and the sum of the pixels obtained by the sum is compared with a predetermined threshold to determine the size and number of the captured object hot zone (specifically, the area to be highlighted). Specifically, the sum of the pixels in the current sliding window is compared with the predetermined threshold, and the area above the predetermined threshold (i.e., the area to be highlighted) is highlighted or colored, that is, the area of the target object (such as an airplane) is highlighted.
[0072] Then, after the sliding traversal operation is completed, a density mask image corresponding to the remote sensing image to be processed can be obtained.
[0073] Further, an evaluation quantity of the object's tilt angle is determined based on the density mask image.
[0074] Specifically, the number of pixels in the density mask image whose pixels are specific pixel values is determined, and the area ratio of the number of pixels in the density mask image whose pixels are specific pixel values to the total number of pixels is calculated using the following expression:
[0075] ;
[0076] in, Indicates the area ratio of the number of pixels having a specific pixel value to the total number of pixels in the density mask image; Indicates the number of pixels in the density mask image whose pixels are specific pixel values; Represents the total number of pixels of all pixels in the density mask image.
[0077] According to the calculated area ratio, the estimated number of tilt angles of the object in the density mask image is determined, for example, by a mask routing prediction module (MRM).
[0078] Specifically, if the density mask image is a black and white binary image, the specific pixel value is 1. In other implementations, if the density mask image is a 256-level grayscale, the pixel value ranges from 0 to 255.
[0079] In a specific embodiment, the number of pixels with a specific pixel value in the density mask image is calculated, and the area ratio of the number of pixels with a pixel value of 1 in the density mask image to the total number of pixels is calculated. After the area ratio is obtained, the area ratio and the number of object inclination angles in Table 1 are used to calculate the area ratio. n The corresponding relationship between the input density mask image and the object tilt angle is obtained. n The specific correspondence is shown in Table 1 below. The correspondence in Table 1 is estimated based on the distribution ratio of target objects in remote sensing images and historical data. n It is not greater than 5 because in conventional remote sensing images, the approximate number of inclination angles of objects does not exceed 5.
[0080] Table 1
[0081]
[0082] Table 1 shows the area ratio and the number of object tilt angles n The corresponding relationship.
[0083] Specifically, the above Table 1 is obtained by comparing a large number of collected remote sensing images (specifically including remote sensing images corresponding to different application tasks such as air traffic management detection tasks, ship supervision detection tasks, adversarial training detection tasks, etc.) and performing data analysis. For example, in the airport scenario, by comparing the distribution characteristics of objects with obvious rotation characteristics (such as aircraft and ships) and the object inclination angle (or rotation angle, specifically the inclination angle or rotation angle of the target object relative to the reference axis or reference object), the statistical analysis experience conclusion that the object inclination angle is less than or equal to 5 is obtained.
[0084] It should be noted that, compared with the area covering fewer objects, the area covering more objects has a higher pixel intensity on the density map. By setting the density pixel threshold, it is possible to estimate the object gathering hotspot and filter out invalid areas covering fewer or no objects. A sliding window is introduced, where the size of the window W is the average overhead area of the possible objects contained in the image. The window is slid with the window size as the step size, and all pixel values in the current window are summed, and the sum is compared with the predetermined threshold. If the sum is lower than the predetermined threshold, all pixel values in the current window are classified as 0. If the sum is higher than or equal to the predetermined threshold, it is classified as 1, and the feature density map after classification is the obtained density map mask. By introducing different predetermined thresholds, the size and number of captured object hotspots can be dynamically adjusted. A smaller threshold will obtain a larger local object hotspot, while increasing the threshold will increase the number of hotspots, but the corresponding hotspot area will be reduced. The contribution of regions of different areas to the rotation feature is generally different, which will lead to the problem of spatial differentiation in the feature extraction process. In addition, the above is only described as an optional example and cannot be understood as a limitation of the present invention.
[0085] The mask branch prediction module obtains the prior information of the object tilt angle from the ratio of the density mask image in the entire feature map, that is, according to the evaluation number of the object tilt angle in the determined density mask image (denoted as n ), further predict the object tilt angle sequence and weight sequence.
[0086] The specific mask branch prediction module performs average pooling on the density mask image after deep convolution encoding, retains more background information of the image while downsampling and reducing the dimension and keeping the features unchanged, so that it can better focus on the tilt angle of the target object, and then more accurately obtains the specific value of the rotation angle of the target object covered in the image and its corresponding weight through the combination layer of linear weighting and nonlinear functions.
[0087] More specifically, the density mask image is input into a lightweight deep convolutional layer with a convolution kernel size of 3*3, and then layer normalization and ReLU activation function are performed. The activated features are passed through an average pooling layer, and the output dimension is C in Next, the pooled feature vector is sent to two different branches, which are processed independently and in parallel. The first branch is the tilt angle (or rotation angle) prediction branch, which consists of a linear layer and a softsign activation function. The number of output channels of the linear layer is determined by the number of estimated tilt angles of the object. n The second branch is the angle weight prediction branch, which consists of a linear layer and a sigmoid activation function. The number of output channels of the linear layer is determined by the number of evaluations of the object's tilt angle. n To avoid learning biased object tilt angles, the linear layer does not learn additive bias. The calculation formula of the softsign layer is as follows.
[0088] ;
[0089] in, SoftSign ( x ) is an activation function with antisymmetric, decentralized and differentiable properties; c is the expansion coefficient, which is used to adjust the rotation search range. Its value range is 0 to 2, but the value may vary depending on the application task. x It is an input feature map matrix, wherein the mask map is obtained after the light convolution layer and the pooling layer (for example, a feature map containing the object tilt angle of the aircraft relative to the reference axis or the reference object, and the number of object tilt angles, a feature map containing the object tilt angle of the ship relative to the reference axis or the reference object, and the number of object tilt angles, and a feature map containing multiple object tilt angles of multiple target objects), and a feature map matrix is further obtained.
[0090] It should be noted that for the value of the expansion coefficient c, in specific practical operations, the value range of c is in the range of 0 to 2. When it is in the range of 0 to 1, it means that the rotation search range is reduced, and when it is in the range of 1 to 2, it means that the rotation search range is increased. For example, in the scenario of air traffic management detection tasks, it is necessary to focus on the object tilt angle or rotation angle of the aircraft. The value of c is in the range of 0 to 1, which can better predict the object tilt angle in the area where the aircraft is distributed in the air traffic management detection task. When there are many types of image objects and many target objects are included in a small area, such as a variety of training equipment, c can be adjusted to 1 to 2 to expand the search range.
[0091] Reference Figure 4 , the feature map of the density mask image passes through the first branch to obtain the object tilt angle sequence, and passes through the second branch to obtain the weight sequence.
[0092] In order to balance the spatial differences caused by contributions from different regions, an adaptive weighted calculation mechanism is proposed to deal with the differences in contributions of rotational features. Figure 4 and Figure 5 , use a set of predicted angles obtained by the mask branch prediction module to perform affine rotation transformation on the convolution kernel, and then perform weighted average processing according to the weights to calculate the weighted combined feature map (including rotation angle orientation, object position, shape, category). This weighted combined feature map improves the network's ability to capture multiple directional object features while still maintaining high efficiency. The preliminary average feature map continues to be input into the backbone for subsequent processing. The details will be explained below.
[0093] Through the deep convolutional layer, the number of evaluations of the tilt angle with the object is adopted n An equal number of convolution kernels are subjected to affine rotation transformation according to the obtained object tilt angle sequence to obtain the transformed updated convolution kernel.
[0094] According to the transformed and updated convolution kernel, the remote sensing image to be processed is subjected to a deep convolution operation and then average pooling, and then a combination layer of linear weighting and nonlinear functions is passed to obtain an object tilt angle sequence and a weight sequence, wherein the weight sequence refers to a sequence formed by weight values corresponding to the tilt angles of each object.
[0095] First, the Spatial-Balance Calculation Module (SCM) constructs the convolutional layer to n (Number of evaluations of the object's tilt angle n The convolution kernel parameters of equal number are taken as input, and the affine transformation operation is performed in combination with the obtained object tilt angle sequence to obtain the convolution kernel parameters after rotation transformation. The rotated convolution kernel parameters are weighted in combination with the obtained weight sequence to obtain the convolution kernel parameters of weighted rotation processing.
[0096] The weighted combination feature map is extracted by n convolution kernels Completed. The size of each convolution kernel is Given a remote sensing image input x to be processed, the rotation angle prediction module obtains a set of rotation angles (Right now Figure 4 The "object tilt angle sequence" shown in Figure 2) and the weights corresponding to each angle (Right now Figure 4 The “weight sequence” shown).
[0097] Specifically, according to the object tilt angle (or rotation angle) sequence , perform affine rotation transformation on n convolution kernels:
[0098] ;
[0099] in, is the convolution kernel updated by the affine rotation transformation; K i is the initial convolution kernel; Each core K i The corresponding object tilt angle (or rotation angle); is an affine transformation operation, i , n All are positive integers.
[0100] It should be noted that the parameters of the initial convolution kernel are regarded as a set of sampling points on the overall two-dimensional plane where the convolution kernel is located. Taking the convolution kernel as an example, nine sampling points are obtained by using bilinear interpolation on the two-dimensional kernel plane. These nine sampling points cover the entire two-dimensional kernel plane in the form of interpolation. Next, the affine transformation matrix corresponding to the obtained object tilt angle sequence is used to rotate the sampling points of the original convolution kernel through the transformation matrix to obtain the sampling points of the convolution kernel after the affine rotation transformation. It is expressed by the following expression, where a and b are the parameters of the original sampling points, representing the coordinates on the two-dimensional plane; a' and b' are the transformed sampling parameters, representing the transformed two-dimensional coordinates, and c is the rotation angle for the rotation transformation:
[0101] ;
[0102] Use n convolution kernels updated by affine rotation transformation to perform convolution operation on the input remote sensing image x to be processed independently, and extract the features of each convolution according to the weight Perform a weighted sum:
[0103] ;
[0104] Among them, y is the output weighted combined feature map, and y is a matrix containing the rotation angle orientation, object position, and shape (category); is the convolution kernel updated by the affine rotation transformation, i , n All are positive integers; Represents the weight corresponding to the object tilt angle sequence, which is used to transform and update the convolution kernel. n is a positive integer; is a convolution operation; x Represents the remote sensing image to be processed.
[0105] Next, in step S104, the remote sensing image to be processed, the obtained object tilt angle sequence and the weight sequence are input into the feature extraction network model to obtain a weighted combined feature map corresponding to the remote sensing image to be processed in different application target detection tasks.
[0106] The convolution operation is performed on the feature map of the input deep convolution layer using a number of convolution kernels equal to the number of object tilt angle assessments. The features extracted by each deep convolution layer are weighted and summed according to the weight sequence:
[0107] ;
[0108] Among them, y represents the weighted combined feature map output by the deep convolution layer, which is a matrix containing the following information: rotation angle, object position, and shape; represents the convolution kernel after transformation and update, n represents the number of convolution kernels, and n is a positive integer; represents a convolution operation; x represents the feature map of the input depth convolution layer.
[0109] In a specific embodiment, the feature extraction network model is, for example, another part independent of the spatial balance calculation module and the mask branch prediction module. For example, it is a single-stage target detection model such as a Resnet network. The second training data set includes remote sensing images annotated with the wings or fuselage of the aircraft, remote sensing images annotated with ships and ports, etc. The feature extraction network model is trained using the second training data set to obtain a trained feature extraction network model.
[0110] The obtained weighted combination feature map is input into the trained feature extraction network model to obtain the weighted combination feature map corresponding to the remote sensing image to be processed in different application target detection tasks.
[0111] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.
[0112] Next, in step S105, target object recognition and positioning matching different application target detection tasks are performed based on the obtained weighted combined feature map to obtain recognition and positioning information.
[0113] In a specific implementation, the obtained weighted combined feature map is input into a detection and recognition model to obtain recognition and positioning information and target object recognition results corresponding to the remote sensing image to be processed.
[0114] Specifically, the detection and recognition model is a single-stage target detection model (such as an S²A-Net model), and a third training data set is specifically established, and the detection and recognition model is trained using the third training data set to obtain a trained detection and recognition model. The third training data set includes remote sensing images annotated with position information of the aircraft relative to the reference axis, position information of the wing, position information of the fuselage, and position information of the runway in the parking area (such as remote sensing images annotated with position information of ships relative to the reference axis in a specific area).
[0115] The trained detection and recognition model is used to detect and recognize the weighted combination feature map to obtain the identification and positioning information. The identified positioning information includes the position information of the aircraft relative to the reference axis in the parking area, the position information of the wing, the position information of the fuselage, the position information of the runway, and the position information of the ship relative to the reference axis in the specific area.
[0116] The target object recognition result includes whether an airplane or a ship is included, the type of the airplane, the type of the ship, the number of target objects, and the object inclination angle of each target object relative to the reference axis.
[0117] In another specific embodiment, for example, the obtained weighted combined feature map including ships and ports is input into a detection and recognition model to obtain the specific position, size, type and bow direction of the ship at the berth in the port area.
[0118] Through the obtained weighted combined feature map, various types of ships (including cargo ships, passenger ships, work vessels, etc.) can be accurately identified and tracked, and their specific location, size, type and bow direction at the berths in the port area can be captured. In fact, being able to capture the above information is crucial for analyzing operational indicators such as port throughput, ship turnover efficiency, and berth utilization. At the same time, by detecting the orientation and distribution of facilities such as cranes, cranes, yards, and warehouses in the port, it is also possible to effectively evaluate the port's loading and unloading capacity and storage conditions, and provide a basis for optimizing cargo scheduling and improving operational efficiency. In addition, by monitoring the dynamics of ships entering and leaving the port and identifying abnormal berthing or aggregation, potential violations can be discovered in a timely manner, thereby improving the effectiveness of maritime safety supervision.
[0119] In another specific implementation, the application target detection task is, for example, an adversarial training detection task, and a weighted combined feature map including adversarial groups and devices in adversarial training is obtained.
[0120] By detecting and identifying remote sensing images of the confrontation training area, it is possible to accurately identify and locate various types of rotating distributed equipment, such as tank formations deployed covertly along ridges, groups of ships parked in ports, aircraft scattered on airport runways, and directional radar equipment set up in complex terrain. The generated density map can quickly highlight high-density target areas, helping analysts quickly identify concentrated deployments or potential assembly points. The predicted object inclination angle helps to determine the tactical intentions of one or the other of the two opposing parties. For example, the direction of movement of a vehicle formation may reveal the route of attack, the formation and orientation of ships can infer their possible navigation plans, and the direction of the radar station can reveal its detection or communication coverage. SCM ensures that in complex environments, whether the target object is in one or more object inclination angles or a rotating state, one or more target objects can achieve feature balanced extraction and accurate identification of location information, which can effectively improve the robustness and accuracy of recognition detection.
[0121] In another example, the remote sensing image to be processed and the obtained weighted combined feature map are input into a detection and recognition model to obtain recognition and positioning information, specifically including a recognition result or a positioning result.
[0122] In another example, the three modules of the adaptive balanced density map inversion network, namely the Density-Map and Mask Generation Module (DGM), the Mask Routing Prediction Module (MRM), and the Spatial-Balance Calculation Module (SCM), are combined with a self-built feature extraction network model or a detection and recognition model to form a new backbone network (backbone), that is, to form a comprehensive model. After the corresponding training process of each module is completed, a trained comprehensive model is obtained.
[0123] The remote sensing image to be processed is input into the trained comprehensive model, and the recognition and positioning information is output, including the recognition result or the positioning result.
[0124] For example, a ship sailing on the sea may point its bow at an arbitrary angle to the horizontal line of the image. Or an airplane at an airport may also present different angles due to different parking positions and shooting angles. The comprehensive model established using the Adaptive Balanced Density Graph Inversion Network (ABD-Net) can accurately capture and locate these target objects with one or more object tilt angles.
[0125] It should be noted that in this example, the three modules of the adaptive balanced density map inversion network, namely the density map mask generation module (Density-Map and Mask Generation Module, DGM), the mask routing prediction module (Mask Routing Prediction Module, MRM) and the spatial balance calculation module (Spatial-Balance Calculation Module, SCM), as well as the self-built feature extraction network model or the detection and recognition model are the same as Figure 1 The relevant contents of the adaptive balanced density map inversion network are roughly the same, so the description of the same parts is omitted. Note that in this example, the weighted combined feature map is used as an intermediate feature map. In addition, in this example, the detection and recognition model is, for example, an S²A-Net model (a single-stage target detection model), which specifically includes a feature pyramid network, a feature alignment module, and a rotation detection module. The S²A-Net model is used to perform classification and regression tasks for target detection, wherein the weighted combined feature map is input as an intermediate variable to the S²A-Net for further processing, and can identify one or more target objects with one or more object inclination angles or arbitrary rotation angles, objects with complex background interference, and objects with obvious differences in contribution from different regions.
[0126] It should be noted that, since the adaptive balanced density map inversion network of the present invention introduces a feature pyramid network to fuse feature maps of different levels, it takes into account the detection of small and large targets, provides multi-scale feature representation, and provides multiple target detection heads, so that at each feature map position, one or more groups of bounding boxes and their corresponding category probabilities and / or object tilt angles are directly predicted. Each bounding box is usually associated with a fixed size and ratio (as well as the case of detecting target objects under the support of multiple object tilt angles). In addition, for multiple detection results of the same category, they are sorted according to, for example, confidence (for example, the confidence is expressed by multiplying the category probability by the bounding box score), and other results that overlap too much with the highest scoring detection result (IOU exceeds a certain threshold) are removed to reduce redundancy and false detection. In addition, for target objects with complex background interference. For example, in an image containing a large number of buildings, vegetation or other complex terrain, to identify ships or aircraft in a specific area, the present invention can highlight the two-dimensional information of the target object through the generated density map, while suppressing background noise, ensuring that the accurate detection of the target is not affected by the surrounding environment, and can accurately identify the target object.
[0127] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.
[0128] Compared with the prior art, the present invention generates a density map corresponding to the remote sensing image to be processed, and generates a density mask image by performing a pixel-level inversion operation on the basis of the density map; determines the evaluation number of the object's tilt angle based on the density mask image, and further predicts the object's tilt angle sequence and weight sequence, and performs feature extraction based on the obtained object tilt angle sequence and weight sequence and the remote sensing image to be processed as the model input of the feature extraction network model, so as to obtain a weighted combination feature map after feature enhancement of the target object, and obtain a weighted combination feature map containing more effective target object features, and then uses the obtained weighted combination feature map for detection and recognition, so as to obtain more accurate target object location information, dynamically predict the tilt angle of the target object in the remote sensing image to be processed, and thus accurately identify target objects with different object tilt angles, and also reduce the amount of calculation in the detection and recognition calculation process.
[0129] In addition, the present invention is particularly suitable for feature balanced extraction and detection and recognition of one or more target objects at an inclined angle or a rotational state in a complex environment, and can effectively improve the robustness and accuracy of recognition and detection.
[0130] Example 2
[0131] The following is an embodiment of the system of the present invention. The method of the first aspect of the present invention is particularly suitable for the remote sensing image processing system based on density mask of the present invention. For details not disclosed in the embodiment of the system of the present invention, please refer to the embodiment of the method of the present invention.
[0132] The second aspect of the present invention provides a remote sensing image processing system based on a density mask, which is used to execute the remote sensing image processing method based on a density mask described in the first aspect of the present invention.
[0133] Reference Figure 6 The remote sensing image processing system 600 based on density mask includes an acquisition processing module 610, a generation processing module 620, a prediction processing module 630, an extraction processing module 640, and an identification and determination module 650.
[0134] Specifically, the acquisition processing module 610 acquires the remote sensing image to be processed that is taken in real time based on different application target detection tasks, and the application target detection tasks include air traffic management detection tasks and ship supervision detection tasks. The generation processing module 620 uses a density map generation network model to generate a density map corresponding to the remote sensing image to be processed. The prediction processing module 630 performs a pixel-level inversion operation on the basis of the generated density map to obtain a density mask image corresponding to the remote sensing image to be processed, determines the evaluation number of the object inclination angle based on the density mask image, and further predicts the object inclination angle sequence and weight sequence. The extraction processing module 640 inputs the remote sensing image to be processed, the obtained object inclination angle sequence and the weight sequence into the feature extraction network model to obtain a weighted combination feature map corresponding to the remote sensing image to be processed in different application target detection tasks. The identification and determination module 650 performs target object identification and positioning matching different application target detection tasks based on the obtained weighted combination feature map to obtain identification and positioning information.
[0135] In an optional embodiment, a deep convolution layer is used to use a number of convolution kernels equal to the number of evaluated object inclination angles, and an affine rotation transformation is performed according to the obtained object inclination angle sequence to obtain a transformed and updated convolution kernel; according to the transformed and updated convolution kernel, the remote sensing image to be processed is subjected to a deep convolution operation and then average pooled, and then passed through a combination layer of linear weighting and nonlinear functions to obtain an object inclination angle sequence and a weight sequence, wherein the weight sequence refers to a sequence formed by weight values corresponding to each object inclination angle.
[0136] In an optional embodiment, the use of a density map generation network model to generate a density map corresponding to the remote sensing image to be processed includes: inputting the remote sensing image to be processed into the density map generation network model, specifically passing through a multi-layer CNN convolutional network and linearly adding to obtain a generation network result, that is, a density map corresponding to the remote sensing image to be processed, and the multi-layer CNN convolutional network includes a three-layer CNN convolutional network.
[0137] In an optional embodiment, the pixel level inversion operation is performed on the basis of the generated density map to obtain a density mask image corresponding to the remote sensing image to be processed, including: performing a sliding traversal operation on the generated density map according to a specified step size by means of a sliding window, and dynamically adjusting the size and number of hot zones of captured objects to obtain a density mask image corresponding to the remote sensing image to be processed.
[0138] In an optional embodiment, the sliding window is adjusted to perform a sliding traversal operation according to different specified step sizes. A sliding traversal operation is performed, that is, a sampling operation is performed. In each sampling operation, all pixel values in the current window are summed to obtain a sum value, and the calculated sum value is compared with a predetermined threshold to determine the size and number of the captured object hot spots.
[0139] The length of the sliding window is within a range of 20 pixels to 50 pixels, the width of the sliding window is within a range of 20 pixels to 50 pixels; and the specified step length is within a range of 5 pixels to 10 pixels.
[0140] In an optional implementation, the remote sensing image to be processed, the obtained object tilt angle sequence and the weight sequence are input into a feature extraction network model to obtain a weighted combined feature map corresponding to the remote sensing image to be processed in different application target detection tasks.
[0141] The convolution operation is performed on the feature map of the input deep convolution layer using a number of convolution kernels equal to the number of object tilt angle assessments. The features extracted by each deep convolution layer are weighted and summed according to the weight sequence:
[0142] ;
[0143] Where y represents the weighted combined feature map output by the deep convolutional layer, and y is a matrix containing the following information: rotation angle, object position, and shape; represents the convolution kernel after transformation and update, n represents the number of convolution kernels, and n is a positive integer; Represents the weight corresponding to the object tilt angle sequence, which is used to transform and update the convolution kernel. n is a positive integer; represents a convolution operation; x represents the feature map of the input depth convolution layer.
[0144] In an optional implementation, the obtained weighted combined feature map is input into a detection and recognition model to obtain recognition and positioning information and target object recognition results corresponding to the remote sensing image to be processed, wherein:
[0145] The detection and recognition model is a single-stage target detection model; the recognition and positioning information includes the position information of the aircraft in the parking area relative to the reference axis, the position information of the wing, the position information of the fuselage, the position information of the runway, and the position information of the ship in the specific area relative to the reference axis; the target object recognition result includes: whether it contains aircraft and ships, the type of aircraft, the type of ship, the number of target objects, and the object inclination angle of each target object relative to the reference axis
[0146] It should be noted that in Example 2, the content of the remote sensing image processing method in the remote sensing image processing system based on density mask of the present invention is roughly the same as the content of the remote sensing image processing method based on density mask in Example 1, so the description of the same parts is omitted.
[0147] Compared with the prior art, the present invention generates a density map corresponding to the remote sensing image to be processed, and generates a density mask image by performing a pixel-level inversion operation on the basis of the density map; determines the evaluation number of the object's tilt angle based on the density mask image, and further predicts the object's tilt angle sequence and weight sequence, and performs feature extraction based on the obtained object tilt angle sequence and weight sequence and the remote sensing image to be processed as the model input of the feature extraction network model, so as to obtain a weighted combination feature map after feature enhancement of the target object, and obtain a weighted combination feature map containing more effective target object features, and then uses the obtained weighted combination feature map for detection and recognition, so as to obtain more accurate target object position information, dynamically predict the tilt angle of the target object in the remote sensing image to be processed, and thus accurately identify target objects with different object tilt angles, and also reduce the amount of calculation in the detection and recognition calculation process.
[0148] In addition, the present invention is particularly suitable for feature balanced extraction and detection and recognition of one or more target objects at an inclined angle or a rotational state in a complex environment, and can effectively improve the robustness and accuracy of recognition and detection.
[0149] Example 3
[0150] Figure 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
[0151] like Figure 7 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.
[0152] The memory stores a computer executable program, which is usually a machine-readable code. The computer executable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps in the method.
[0153] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).
[0154] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.
[0155] It should be understood that Figure 7 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.
[0156] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 8 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.
[0157] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0158] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0159] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0160] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.
[0161] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.
[0162] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.
[0163] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.
Claims
1. A remote sensing image detection and processing method based on density mask, characterized in that: include: Based on different application target detection tasks, remote sensing images to be processed that are taken in real time are obtained, and the application target detection tasks include air traffic management detection tasks and ship supervision detection tasks; A density map generation network model is used to generate a density map corresponding to the remote sensing image to be processed; A pixel-level inversion operation is performed on the basis of the generated density map, and a sliding traversal operation is performed on the generated density map in a sliding window manner according to a specified step size, and the size and number of the captured object hot zones are dynamically adjusted to obtain a density mask image corresponding to the remote sensing image to be processed. Specifically, a sliding traversal operation is performed by adjusting the sliding window according to different specified step sizes. A sliding traversal operation is performed once, i.e., a sampling operation is performed; In each sampling operation, a sum operation is performed on all pixel values in the current window to obtain a sum value, and the calculated sum value is compared with a predetermined threshold value to determine the size and number of the captured object hot zone; based on the density mask image, the evaluation number of the object tilt angle is determined, specifically the number of pixel points with specific pixel values in the density mask image is determined, and the following expression is used to calculate the area ratio of the number of pixel points with specific pixel values in the density mask image to the total number of pixel points: ; in, Indicates the area ratio of the number of pixels with a specific pixel value in the density mask image to the total number of pixels; pixel mask Indicates the number of pixels with a specific pixel value in the density mask image; pixel DesityMap Representing the total number of pixels of all pixels in the density mask image; determining the estimated number of object inclination angles in the density mask image according to the calculated area ratio; and further predicting the object inclination angle sequence and weight sequence, specifically through a deep convolution layer, using a number of convolution kernels equal to the estimated number of object inclination angles, performing an affine rotation transformation according to the obtained object inclination angle sequence, and obtaining a transformed and updated convolution kernel; according to the transformed and updated convolution kernel, performing average pooling on the remote sensing image to be processed after a deep convolution operation, and then obtaining an object inclination angle sequence and a weight sequence through a combination layer of linear weighting and nonlinear functions, wherein the weight sequence refers to a sequence formed by weight values corresponding to each object inclination angle; The remote sensing image to be processed, the obtained object tilt angle sequence and the weight sequence are input into the feature extraction network model to obtain a weighted combination feature map corresponding to the remote sensing image to be processed in different application target detection tasks; According to the obtained weighted combined feature map, target object recognition and positioning matching different application target detection tasks are performed to obtain recognition and positioning information.
2. The remote sensing image detection and processing method based on density mask according to claim 1, characterized in that: The method of using a density map generation network model to generate a density map corresponding to the remote sensing image to be processed includes: The remote sensing image to be processed is input into a density map generation network model, and specifically, a generation network result is obtained after a multi-layer CNN convolutional network and linear addition, that is, a density map corresponding to the remote sensing image to be processed, and the multi-layer CNN convolutional network includes a three-layer CNN convolutional network.
3. The remote sensing image detection and processing method based on density mask according to claim 1, characterized in that: The length of the sliding window is within a range of 20 pixels to 50 pixels, the width of the sliding window is within a range of 20 pixels to 50 pixels; and the specified step length is within a range of 5 pixels to 10 pixels.
4. The remote sensing image detection and processing method based on density mask according to claim 1, characterized in that: The remote sensing image to be processed, the obtained object tilt angle sequence and the weight sequence are input into the feature extraction network model to obtain the weighted combination feature map corresponding to the remote sensing image to be processed in different application target detection tasks, including: The convolution operation is performed on the feature map of the input deep convolution layer using a number of convolution kernels equal to the number of object tilt angle assessments, and the features extracted by each deep convolution layer are weighted and summed according to the weight sequence: ; Where y represents the weighted combined feature map output by the deep convolutional layer, and y is a matrix containing the following information: rotation angle, object position, and shape; represents the convolution kernel after transformation and update, n represents the number of convolution kernels, and n is a positive integer; Represents the weight corresponding to the object tilt angle sequence, which is used to transform and update the convolution kernel. n is a positive integer; Represents a convolution operation; x represents the feature map of the input depth convolution layer.
5. The remote sensing image detection and processing method based on density mask according to claim 1, characterized in that: The obtained weighted combined feature map is input into the detection and recognition model to obtain the recognition and positioning information and target object recognition results corresponding to the remote sensing image to be processed, where: The detection and recognition model is a single-stage target detection model; the recognition and positioning information includes the position information of the aircraft in the parking area relative to the reference axis, the position information of the wing, the position information of the fuselage, the position information of the runway, and the position information of the ship in the specific area relative to the reference axis; the target object recognition result includes: whether an aircraft or a ship is included, the type of aircraft, the type of ship, the number of target objects, and the object inclination angle of each target object relative to the reference axis.
6. A remote sensing image detection and processing system based on density mask, which is used to execute the remote sensing image detection and processing method based on density mask according to any one of claims 1 to 5, characterized in that: The remote sensing image detection and processing system comprises: An acquisition processing module acquires the remote sensing images to be processed that are taken in real time based on different application target detection tasks, wherein the application target detection tasks include air traffic management detection tasks and ship supervision detection tasks; A generation processing module uses a density map generation network model to generate a density map corresponding to the remote sensing image to be processed; The prediction processing module performs a pixel-level inversion operation on the basis of the generated density map, performs a sliding traversal operation on the generated density map in a sliding window manner according to a specified step size, and dynamically adjusts the size and number of the hot areas of the captured object to obtain a density mask image corresponding to the remote sensing image to be processed. Specifically, the sliding traversal operation is performed by adjusting the sliding window according to different specified step sizes, and a sliding traversal operation is performed, that is, a sampling operation is performed; in each sampling operation, all pixel values in the current window are summed to obtain a sum value, and the calculated sum value is compared with a predetermined threshold to determine the size and number of the hot areas of the captured object; based on the density mask image, the evaluation number of the object tilt angle is determined, specifically the number of pixel points in the density mask image whose pixels are specific pixel values is determined, and the following expression is used to calculate the area ratio of the number of pixel points in the density mask image whose pixels are specific pixel values to the total number of pixel points: ; in, Indicates the area ratio of the number of pixels with a specific pixel value in the density mask image to the total number of pixels; pixel mask Indicates the number of pixels with a specific pixel value in the density mask image; pixel DesityMap Representing the total number of pixels of all pixels in the density mask image; determining the estimated number of object inclination angles in the density mask image according to the calculated area ratio; and further predicting the object inclination angle sequence and weight sequence, specifically through a deep convolution layer, using a number of convolution kernels equal to the estimated number of object inclination angles, performing an affine rotation transformation according to the obtained object inclination angle sequence, and obtaining a transformed and updated convolution kernel; according to the transformed and updated convolution kernel, performing average pooling on the remote sensing image to be processed after a deep convolution operation, and then obtaining an object inclination angle sequence and a weight sequence through a combination layer of linear weighting and nonlinear functions, wherein the weight sequence refers to a sequence formed by weight values corresponding to each object inclination angle; The extraction processing module inputs the remote sensing image to be processed, the obtained object tilt angle sequence and the weight sequence into the feature extraction network model to obtain the weighted combination feature map corresponding to the remote sensing image to be processed in different application target detection tasks; The recognition and determination module performs target object recognition and positioning matching different application target detection tasks based on the obtained weighted combined feature map to obtain recognition and positioning information.
