A target detection method based on multi-scale deformation modeling and region fine extraction

Abstract

A target detection method based on multi-scale deformation modeling and regional fine extraction belongs to the field of target detection and recognition technology. The method is as follows: design a multi-scale feature dynamic fusion module to obtain the candidate region extraction results of the multi-scale deformation target; classify the candidate region extraction results to obtain all target regions marked with the minimum bounding rectangular box; propose a customized significant region fine extraction loss function combining binary cross entropy, bias loss and structural similarity to achieve accurate detection of ship targets under complex scene interference. The present invention realizes enhanced learning of multi-scale deformation features of targets, improves the detection network's ability to distinguish targets from complex backgrounds, and can achieve accurate detection of ship targets in complex scenes such as clouds, light shadows, port facilities, etc., providing strong support for military and civilian applications such as air traffic control, maritime rescue and port management.

Description

Technical Field

[0001] The invention relates to a target detection method based on multi-scale deformation modeling and region fine extraction, belonging to the technical field of target detection and recognition. Background Art

[0002] Ship detection technology in optical remote sensing images plays a vital role in many military and civilian applications such as maritime rescue, port management, and sea surveillance. Visible light remote sensing images have rich structural information and texture features, which conform to the visual observation habits of the human eye and have become an important data source for ship detection. However, due to the long-distance and random direction observation characteristics of remote sensing detection, the size and shape of ships in optical remote sensing images may vary to varying degrees, resulting in diverse differences in target features, which puts higher requirements on the robustness of the detection model. In addition, due to the wide-area imaging characteristics of optical remote sensing, the complex scene interference such as clouds, light shadows, and port facilities that may exist in remote sensing images further increases the difficulty of the ship detection model. Therefore, it is still a challenging task to achieve effective detection of ship targets under the conditions of complex optical scenes and variable target features.

[0003] With the development of deep learning technology, deep learning architectures represented by convolutional neural networks have provided an effective framework for ship detection in remote sensing images with their excellent high-level semantic representation advantages. In order to enable the network to effectively detect ship targets of different scales and types, the Feature Pyramid Network (FPN) has been widely used to solve the problem of multi-scale target detection in images because of its advantage of representing target features at different levels.

[0004] The multi-level structure of FPN actually uses information from different receptive fields. By mining the contextual relationship between the target and the background in different receptive fields, the detection model can learn the differential characteristics of the target and the background at different scales, thereby ultimately achieving effective separation of the target and the background. However, due to the multi-factor coupled changes in the scale and deformation of the target in remote sensing detection, it is clear that not all receptive field pixels contribute to the response of the network output unit to the target detection task. How to dynamically select different receptive fields and perform information fusion to achieve effective learning of the multi-scale and deformation characteristics of the target is one of the issues that need to be studied in the field of ship detection. In addition, due to the inherent locality of the convolution operation and the fact that the commonly used loss functions (such as the cross entropy loss function) only rely on the characteristics of local measurements, the current deep learning detection model is difficult to achieve uniform and fine saliency extraction of objects, resulting in incomplete saliency areas of ships in complex scenes, which limits the performance of the detection algorithm in detecting targets under conditions such as complex ports and cloud interference.

[0005] Therefore, there is an urgent need to accurately model and characterize the multi-scale, multi-receptive field and deformation characteristics of ship targets in optical remote sensing images, and enable the model to learn accurate and comprehensive differential features between the target and the background, so as to achieve effective detection of ship targets in complex scenes. Summary of the invention

[0006] In order to solve the problems of multi-scale, multi-deformation, and complex scene interference of ship targets in remote sensing images in the background technology, the present invention provides a ship detection method based on multi-scale deformation characteristic modeling and fine extraction of salient regions. By constructing a multi-receptive field deformation characteristic mining module and a multi-scale deformation feature enhancement module, the multi-scale deformation characteristics of ship targets are characterized and modeled, and the global and local features such as brightness, structure, and contrast are comprehensively used to construct loss function constraints, improve the fine extraction capability of salient regions, and achieve effective detection of ship targets in various complex scenes, which can provide strong support for military and civilian applications such as air traffic control, maritime rescue, and port management.

[0007] To achieve the above object, the present invention adopts the following technical solution: a target detection method based on multi-scale deformation modeling and region fine extraction, the method comprising the following steps:

[0008] S1: Design a multi-scale feature dynamic fusion module to obtain the candidate region extraction results of multi-scale deformable targets;

[0009] S2: The candidate region extraction results obtained in S1 are input into the classic R0CNN structure for classification to obtain all target regions marked with the minimum bounding rectangular box; a customized salient region fine extraction loss function combining binary cross entropy, bias loss and structural similarity is proposed to achieve accurate detection of ship targets under complex scene interference.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] 1. Aiming at the problems of difficult multi-scale characterization and diverse deformation characteristics of ship targets, the present invention designs a saliency extraction network that jointly models the multi-scale and deformation characteristics of the target. Multi-branch dilated convolution and deformable convolution are used to comprehensively learn the difference characteristics between the target and the background and the deformation characteristics of the target itself. Feature descriptors are proposed from the two aspects of channel dimension and spatial dimension to dynamically optimize the features extracted by multiple receptive field branches, realize enhanced learning of multi-scale deformation characteristics of the target, improve the detection network's ability to distinguish between the target and the complex background, and realize accurate detection of ship targets in complex scenes such as clouds, light shadows, port facilities, etc., providing strong support for military and civilian applications such as air traffic control, maritime rescue and port management.

[0012] 2. Aiming at the problem of incomplete salient areas of ship targets under the interference of complex scenes, the present invention proposes a fine target area highlight loss function, which comprehensively considers the brightness, contrast and structural characteristics of ship targets, constrains the predicted salient areas from both global and local aspects, thereby improving the accuracy of fine extraction of target edge contours and improving the ship detection performance of the detection model in complex scenes such as port facilities and cloud interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the present invention;

[0014] Figure 2 It is a schematic diagram of the structure of the multi-receptive field deformation feature mining module and the multi-scale deformation feature enhancement module;

[0015] Figure 3 It is a comparison chart of the original image of the dataset and the significant feature extraction results of the corresponding typical ship target;

[0016] Figure 4 This is a typical ship target detection result image of the dataset image. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0018] A target detection method based on multi-scale deformation modeling and region fine extraction, the method comprising the following steps:

[0019] S1: Use dilated convolution and variable convolution with different dilation rates to construct multi-scale feature extraction branches, explore and characterize the deformation characteristics of multiple receptive fields, design a multi-scale feature dynamic fusion module, generate channel information density statistical weights and spatial scale feature descriptors by compressing channel dimensions and spatial dimensions, adjust the weights of features from multiple receptive field branches, enhance the semantic information and edge details of the target in the feature map, and obtain the candidate region extraction results of multi-scale deformable targets;

[0020] S101: Using the typical ResNet0101 network as the backbone feature extraction network, the input remote sensing image is extracted layer by layer with multi-level feature maps of downsampling levels of 2, 4, 8, 16 and 32 times, which are denoted as F1, F2, F3, F4 and F5 respectively;

[0021] S102: The multi-receptive field deformation feature mining module (MDCMM) is used to extract the feature information of different receptive fields (realized by dilated convolutions with different dilation rates) in the feature map of the same branch to better capture the scale change of the target. At the same time, the deformable convolution network and multi-receptive field branches are comprehensively used to improve the mining ability of multi-scale deformation characteristics of the target area. The network diagram is shown in the figure below. Figure 2 As shown in the figure. This module uses a skip connection structure to provide additional multi-scale and multi-level features, reduce the compression loss of image information, aggregate multi-scale features through channel cascading and 1×1 convolution, and restore the same number of channels as the input feature map. This module mainly includes 6 3×3 convolution layers, 1 1×1 convolution layer, and 2 3×3 deformable convolution layers.

[0022] S10201: Let the input feature map be F n , extract the feature map F by dilation rate m of 3, 6, and 9 respectively n The multi-scale features in are as follows:

[0023] F n =conv(concat(AC m =3(F n ),AC m =6(F n ),AC m =9(F n ),conv(F n ))) (1)

[0024] In formula (1):

[0025] AC m (·) represents a dilated convolution with a dilation rate of m;

[0026] concat(·) represents channel concatenation operation;

[0027] conv(·) represents a 1×1 convolutional layer;

[0028] F n It represents the feature map obtained after the dilation rate m of the hole convolution processing in the multi-receptive field deformation feature mining module;

[0029] S10202: In order to adapt to the variable shape of the target, after extracting multi-scale features, deformable convolution is introduced to automatically adjust the shape change modeling. Compared with ordinary convolution, deformable convolution additionally learns offset coordinate information, which can dynamically refine the receptive field during the convolution process. The intermediate input feature map is calculated at the initial center coordinate p0 of the convolution kernel through deformable convolution. The deformable convolution feature of is as follows:

[0030]

[0031] In formula (2):

[0032] R = {(-1,-1), (-1,0), ..., (0,-1), (1,1)} represents the relative position of the convolution kernel;

[0033] w represents weight;

[0034] p n Represents the index of the convolution kernel;

[0035] Δp n Represents the bias coordinates of the convolution kernel.

[0036] Represents the intermediate input feature map Feature map enhanced by the multi-receptive field deformation feature mining module;

[0037] p0 represents the initial center coordinate of the convolution kernel.

[0038] S103: After using the multi-receptive field deformation feature mining module, in order to further aggregate the multi-scale information of the target and improve the detection network's ability to distinguish between the target and the complex background, a multi-scale feature dynamic fusion module (MSDFM) is designed to control the information flow from multiple receptive field branches by compressing the channel dimension and the spatial dimension, and adjust the weights of the fusion features enhanced by the multi-receptive field deformation feature mining module to enhance the semantic information, edge details and global correlation of the feature map;

[0039] S10301: The feature map enhanced by the multi-receptive field deformation feature mining module As the input feature map and the original feature map output by the backbone network Perform the sum operation to obtain the initial fused feature map The formula is as follows:

[0040]

[0041] S10302: Use global average pooling to embed global information, generate channel information density statistical weights, and perform preliminary fusion on the feature map The weights are redistributed to enhance effective features and suppress irrelevant noise interference. The formula is as follows:

[0042]

[0043] In formula (4):

[0044] H represents the height of the initially fused feature map;

[0045] W represents the width of the initially fused feature map;

[0046] i represents the height position index of the feature point in the initially fused feature map;

[0047] j represents the width position index of the feature point in the initial fused feature map;

[0048] S c Represents the cth element in the density statistics weight;

[0049] S10303: Use the fully connected layer to transform the cth element S in the density statistical weight c It is mapped into a compact feature descriptor, which enables accurate multi-branch adaptive selection with good computational efficiency and suppresses scene false alarm interference. The formula is as follows:

[0050] z=δ(BN(FC(S c ))) (5)

[0051] In formula (5):

[0052] δ represents the ReLU activation function;

[0053] BN represents batch normalization operation;

[0054] FC represents the fully connected layer;

[0055] z represents the feature descriptor of the multi-scale information space;

[0056] S10304: A cross-channel soft attention mechanism is used to guide the model to adaptively select different information space scales, that is, different receptive field branches, through the feature descriptor z. The weight calculation formula of different branches is as follows:

[0057]

[0058] In formula (6):

[0059] A and B represent the soft attention transformation matrices of the feature descriptor z on two different branches respectively;

[0060] A c and B c Respectively represent the c-th row corresponding to the two transformation matrices;

[0061] a and b represent the dynamic weights of two different branches respectively;

[0062] a c and b c Respectively represent the cth element corresponding to the two dynamic weights, and a c +b c =1;

[0063] S10305: Use dynamic weights to combine the initial fused feature maps And the original feature map output by the backbone network The final fusion is performed to obtain the feature map of the final output of the multi-scale deformation characteristic modeling network The formula is as follows:

[0064]

[0065] S104: Input the feature map into the Region Proposal Network (RPN) to generate a target candidate region, and obtain a candidate region extraction result of a multi-scale deformable target.

[0066] S2: The candidate region extraction results obtained in S1 are input into the classic R0CNN structure for classification to obtain all target areas marked with the minimum bounding rectangular box; in order to improve the overall classification performance of the model, the brightness, contrast and structural characteristics of the ship are comprehensively considered, and a customized salient region fine extraction loss function combining binary cross entropy, bias loss and structural similarity is proposed to achieve accurate detection of ship targets under the interference of complex scenes such as port facilities and cloud interference.

[0067] S201: For salient ship detection, we first use the binary cross entropy (BCE) loss function to measure the pixel difference between the extracted predicted salient area and the true labeled area. The binary cross entropy loss function L BCE The definition is as follows:

[0068]

[0069] In formula (8):

[0070] P represents the predicted significant region;

[0071] G represents the real marked area;

[0072] p represents the probability of belonging to the predicted salient region

[0073] g represents the probability of belonging to the true labeled area;

[0074] S202: In order to improve the overlap probability between the predicted foreground area and the true value, the intersection-over-union relationship between the predicted salient area P and the true annotated area G is reconstructed in the form of prediction probability. In order to ensure the monotonicity of the loss function constraint, the overlap probability is converted into a bias loss function L Offset , the formula is as follows:

[0075]

[0076] S203: Binary cross entropy loss function L BCEAnd the bias loss function L Offset The extracted salient regions are constrained mainly from a structural perspective. Considering that the structural similarity index can comprehensively represent the properties of the image from three aspects: brightness, contrast, and structure, assuming that the predicted salient region corresponds to a pixel set x, and the real annotated region corresponds to a pixel set y, the structural similarity loss function L SSIM The definition is as follows:

[0077]

[0078] In formula (10):

[0079] μ x Represents the average value of the predicted salient area pixel set;

[0080] μ y Represents the average value of the pixel set in the real annotation area;

[0081] σ x Represents the standard deviation of the predicted salient region pixel set;

[0082] σ y Represents the standard deviation of the pixel set in the true annotation area;

[0083] σ xy represents covariance;

[0084] c1 and c2 are constants to prevent the denominator from being zero, and c1 = 0.01, c2 = 0.03;

[0085] S204: Binary cross entropy loss is calculated based on local pixel positions, and bias loss is calculated for the global area. Considering the limitations of brightness and contrast on the detection ability of the model, the final loss function L IOU The definition is as follows:

[0086] L IOU =αL BCE +βL Offset +γL SSIM (11)

[0087] In formula (11):

[0088] β and γ are weight parameters.

[0089] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0090] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A target detection method based on multi-scale deformation modeling and region fine extraction, characterized in that: The method comprises the following steps: S1: Design a multi-scale feature dynamic fusion module to obtain the candidate region extraction results of multi-scale deformable targets; S2: Input the candidate region extraction results obtained in S1 into the classic R0CNN structure for classification, and obtain all target regions marked with the minimum bounding rectangle; propose a customized salient region fine extraction loss function that combines binary cross entropy, bias loss and structural similarity to achieve accurate detection of ship targets under complex scene interference; The S2 comprises the following steps: S201: Use the binary cross entropy loss function to measure the pixel difference between the extracted predicted salient area and the true labeled area. The binary cross entropy loss function L BCE The definition is as follows: In formula (8): P represents the predicted significant region; G represents the real marked area; p represents the probability of belonging to the predicted salient region g represents the probability of belonging to the true labeled area; S202: In order to improve the overlap probability between the predicted foreground area and the true value, the intersection-over-union relationship between the predicted salient area P and the true annotated area G is reconstructed in the form of prediction probability. In order to ensure the monotonicity of the loss function constraint, the overlap probability is converted into a bias loss function L Offset , the formula is as follows: S203: Considering the structural similarity index, assuming that the predicted salient region corresponds to a pixel set x, and the real annotated region corresponds to a pixel set y, then the structural similarity loss function L SSIM The definition is as follows: In formula (10): μ x Represents the average value of the predicted salient area pixel set; μ y Represents the average value of the pixel set in the real annotation area; σ x Represents the standard deviation of the predicted salient region pixel set; σ y Represents the standard deviation of the pixel set in the true annotation area; σ xy represents covariance; c1 and c2 are constants to prevent the denominator from being zero, and c1 = 0.01, c2 = 0.03; S204: Comprehensively considering the limitations of brightness and contrast on the model detection ability, the final loss function L IOU The definition is as follows: L IOU =αL BCE +βL Offset +γL SSIM (11) In formula (11): β and γ are weight parameters.

2. The target detection method based on multi-scale deformation modeling and region fine extraction according to claim 1, characterized in that: The S1 comprises the following steps: S101: extracting multi-level feature maps layer by layer from the input remote sensing image, which are denoted as F1, F2, F3, F4 and F5 respectively; S102: extracting feature information of different receptive fields in the feature map of the same branch using a multi-receptive field deformation feature mining module; S103: Design a multi-scale feature dynamic fusion module to control the information flow from multiple receptive field branches by compressing the channel dimension and spatial dimension, and adjust the weights of the fusion features enhanced by the multi-receptive field deformation feature mining module to enhance the semantic information, edge details and global correlation of the feature map; S104: Input the feature map into a region proposal network to generate a target candidate region, and obtain a candidate region extraction result of a multi-scale deformable target.

3. The target detection method based on multi-scale deformation modeling and region fine extraction according to claim 2 is characterized in that: The S102 comprises the following steps: S10201: Extract feature map F n The multi-scale features in are as follows: Fn=conv(concat(ACm=3(Fn),ACm=6(Fn),ACm=9(Fn),conv(Fn))) (1) In formula (1): AC m (·) represents a dilated convolution with a dilation rate of m; concat(·) represents channel concatenation operation; conv(·) represents a 1×1 convolutional layer; F n It represents the feature map obtained after the dilation rate m of the hole convolution processing in the multi-receptive field deformation feature mining module; S10202: Introduce deformable convolution to automatically adjust shape change modeling, and use deformable convolution to calculate the intermediate input feature map at the initial center coordinate p0 of the convolution kernel The deformable convolution feature of is as follows: In formula (2): R = {(-1,-1), (-1,0), ..., (0,-1), (1,1)} represents the relative position of the convolution kernel; w represents weight; p n Represents the index of the convolution kernel; Δp n Represents the bias coordinates of the convolution kernel; Represents the intermediate input feature map Feature map enhanced by the multi-receptive field deformation feature mining module; p0 represents the initial center coordinate of the convolution kernel.

4. The target detection method based on multi-scale deformation modeling and region fine extraction according to claim 3 is characterized in that: The S103 comprises the following steps: S10301: The feature map enhanced by the multi-receptive field deformation feature mining module As the input feature map and the original feature map output by the backbone network Perform the sum operation to obtain the initial fused feature map The formula is as follows: S10302: Use global average pooling to embed global information, generate channel information density statistical weights, and perform preliminary fusion on the feature map The weights are redistributed to enhance effective features and suppress irrelevant noise interference. The formula is as follows: In formula (4): H represents the height of the initially fused feature map; W represents the width of the initially fused feature map; i represents the height position index of the feature point in the initially fused feature map; j represents the width position index of the feature point in the initial fused feature map; S c Represents the cth element in the density statistics weight; S10303: Use the fully connected layer to transform the cth element S in the density statistical weight c It is mapped into a compact feature descriptor, which enables accurate multi-branch adaptive selection and suppresses scene false alarm interference. The formula is as follows: z=δ(BN(FC(S c ))) (5) In formula (5): δ represents the ReLU activation function; BN represents batch normalization operation; FC represents the fully connected layer; z represents the feature descriptor of the multi-scale information space; S10304: A cross-channel soft attention mechanism is used to guide the model to adaptively select different information space scales, that is, different receptive field branches, through the feature descriptor z. The weight calculation formula of different branches is as follows: In formula (6): A and B represent the soft attention transformation matrices of the feature descriptor z on two different branches respectively; A c and B c Respectively represent the c-th row corresponding to the two transformation matrices; a and b represent the dynamic weights of two different branches respectively; a c and b c Respectively represent the cth element corresponding to the two dynamic weights, and a c +b c =1; S10305: Use dynamic weights to combine the initial fused feature maps And the original feature map output by the backbone network The final fusion is performed to obtain the feature map of the final output of the multi-scale deformation characteristic modeling network The formula is as follows:

Images