Methods, apparatus, and equipment for multi-scale target detection in remote sensing images based on cross-fusion detection heads

Abstract

This disclosure provides a method, apparatus, and device for multi-scale target detection in remote sensing images based on a cross-fusion detection head. It can be applied to the fields of image technology and target detection technology. The method includes: extracting multiple sets of scale feature maps from the original image using a feature extraction module; inputting the multiple sets of scale feature maps into a feature fusion module to obtain a fused feature map; processing the fused feature map using multiple branches in the cross-fusion detection head; and obtaining a target recognition result by cross-fusion of the processing results from each branch. The cross-fusion detection head includes multiple branches, and different branches output different processing results. By cross-fusioning information from classification and regression branches using the cross-fusion detection head, the information barrier between the classification and regression branches is broken down, enriching semantic information while enhancing feature representation, effectively improving target detection accuracy.

Description

Technical Field

[0001] This disclosure relates to the fields of image technology and target detection technology, and more specifically, to a method, apparatus and device for multi-scale target detection in remote sensing images based on a cross-fusion detection head. Background Technology

[0002] Remote sensing target monitoring is of great significance in information gathering, disaster relief, and real-time target tracking. Traditional remote sensing image target detection methods rely on manually designed features, which are difficult to adapt to interference from complex backgrounds and variable atmospheric environments.

[0003] In recent years, with the development of high-resolution optical satellite imaging technology and artificial intelligence, target detection, as a classic research direction in image processing, has also faced new challenges and breakthroughs. Some scholars have applied deep learning to target detection tasks in remote sensing images. Targets in remote sensing images often exhibit diverse scales and complex backgrounds, posing significant challenges to accurate detection. The detection head is a crucial module connecting feature extraction and the final detection result. Its design and optimization directly affect the performance of the target detection network, including detection accuracy and adaptability to different scales and complex backgrounds.

[0004] In existing two-stage object detection networks, the detection head typically consists of two branches: a classification branch and a regression branch. These branches independently provide object classification and location information. In the classification branch, the region surrounding the object in the feature map remains active, while non-object regions gradually approach zero. Similarly, the regression branch contains scale information of the captured object; different objects exhibit different scale distributions. Existing object detection networks that only utilize information from one branch may lead to insufficient utilization of the location information from the classification branch and the classification information from the regression branch. This results in problems such as information loss and inaccurate identification, leading to low detection accuracy when performing multi-scale object detection on remote sensing images. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] This disclosure provides a method, apparatus, and electronic device for multi-scale target detection in remote sensing images based on a cross-fusion detection head, which is used to at least partially solve one of the above-mentioned technical problems.

[0007] (II) Technical Solution

[0008] According to a first aspect of this disclosure, a method for multi-scale target detection in remote sensing images based on a cross-fusion detection head is provided, comprising: extracting multiple sets of scale feature maps from the original image based on a feature extraction module; inputting the multiple sets of scale feature maps into a feature fusion module to obtain a fused feature map based on the multiple sets of scale feature maps; processing the fused feature map based on multiple branches in the cross-fusion detection head, and obtaining a target recognition result by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and the processing results output by different branches are different.

[0009] According to embodiments of this disclosure, the cross-fusion detection head includes a region candidate structure and a hybrid rotating box generation structure. It processes fusion information based on branches in the cross-fusion detection head, and obtains a target recognition result by fusing the processing results of each branch. The process includes: inputting a fusion feature map into the region candidate structure; processing the fusion feature map using different first branches in the region candidate structure and fusing the processing results of each branch to obtain first information; inputting the first information into the hybrid rotating box generation structure; processing the first information using different second branches in the hybrid rotating box generation structure and cross-fusing the processing results of each second branch to obtain a target recognition result.

[0010] According to embodiments of this disclosure, the region candidate structure includes a first branch module and a first branch fusion module. The first branch module includes a first classification branch unit and a first regression branch unit. A fused feature map is input into the region candidate structure, and different first branches in the region candidate structure process the fused feature map and fuse the processing results of each branch to obtain first information. This includes: inputting the fused feature map into the first classification branch unit and the first regression branch unit respectively; the first classification branch unit enhances the regression details of pixels near the target object and suppresses the regression of non-target regions; and the first regression branch captures the scale information of the target object. The regression details and scale information are input into the first branch fusion module, and the first information is determined based on the first branch fusion module. The first information includes target bounding box information and the probability of the target object's existence.

[0011] According to an embodiment of this disclosure, regression detail and scale information are input to a first branch fusion module, and first information is determined based on the first branch fusion module, including: convolving the regression detail and scale information to unify the number of channels of the regression detail and scale information; and fusing the unified regression detail and scale information with the initial channel information to obtain the first information.

[0012] According to embodiments of this disclosure, the hybrid rotating bounding box generation structure includes a shared fully convolutional module, a second branch module, and a second branch fusion module. The second branch module includes a second classification branch unit and a second regression branch unit. Inputting first information into the hybrid rotating bounding box generation structure, processing the first information using different second branches within the structure, and cross-fusing the processing results of each second branch to obtain a target recognition result includes: inputting the first information output from the region candidate structure into the shared fully convolutional module; performing a shared convolution on the first information using the shared fully convolutional module; inputting the convolutionally processed first information into the second regression branch and the second classification branch respectively; processing the input first information using the second regression branch unit and the second classification branch unit respectively to obtain temporary location information and temporary category information; and cross-fusing the temporary location information, temporary category information, and the first information using the second branch fusion module to determine the target recognition result, wherein the target recognition result includes target category information and target location information.

[0013] According to embodiments of this disclosure, multiple sets of feature maps are extracted from the original image based on the feature extraction module, including: performing feature extraction on the original image based on different extraction branches to obtain multiple sets of feature maps; wherein, different extraction branches use different networks; and converting and fusing the multiple sets of feature maps based on different channels to obtain multiple sets of scale feature maps.

[0014] According to embodiments of this disclosure, multiple sets of scale feature maps are input to a feature fusion module to obtain fusion information, including: performing the following operations on each scale feature map: upsampling the current scale feature map using bilinear interpolation and deconvolution to match the size of the current scale feature map with the next scale feature map; adding the upsampled feature map to the feature map of the corresponding scale to obtain the fused feature map corresponding to the current scale feature map; repeating the above process until the fused feature maps corresponding to all scale feature maps are obtained.

[0015] According to embodiments of this disclosure, the method further includes: mapping the target recognition result onto the original image, and displaying the target recognition result in the form of a detection box and the detection result.

[0016] According to a second aspect of this disclosure, a remote sensing image multi-scale target detection device based on a cross-fusion detection head is provided, comprising: a feature extraction module for extracting multiple sets of scale feature maps from the original image; a feature fusion module for obtaining a fused feature map based on the multiple sets of scale feature maps; and a cross-fusion detection head for performing branch processing on the fused feature map and obtaining a target recognition result by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and the processing results output by different branches are different.

[0017] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0018] (III) Beneficial Effects

[0019] The remote sensing image multi-scale target detection method, apparatus, and electronic equipment based on cross-fusion detection head disclosed herein have at least the following beneficial effects:

[0020] A cross-fusion detection head is employed to effectively break down the information barrier between the classification and regression branches in the detection head. By cross-summing the outputs of the regression and classification branches, more hidden information is obtained. The classification branch can effectively enhance the regression details of pixels near the target and suppress the regression of non-target regions, thus significantly improving the accuracy of remote sensing target detection. Furthermore, randomly generated proposal candidate boxes are used to refine the generated data and overlay the predictions of changes, thereby further improving the accuracy of remote sensing target detection and the precision of target detection in remote sensing images. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 A flowchart illustrating a multi-scale target detection method for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure is shown.

[0023] Figure 2 This diagram schematically illustrates the structure of a feature extraction module according to an embodiment of the present disclosure.

[0024] Figure 3 This diagram schematically illustrates the structure of a feature fusion module according to an embodiment of the present disclosure;

[0025] Figure 4 This schematic diagram illustrates a structural schematic of a region candidate structure according to an embodiment of the present disclosure;

[0026] Figure 5 This schematic diagram illustrates the structure of the hybrid rotating frame generation structure according to an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram illustrates the structure of a multi-scale target detection device for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure;

[0028] Figure 7A block diagram schematically illustrates an electronic device suitable for implementing a multi-scale target detection method for remote sensing images based on a cross-fusion detection head, according to embodiments of the present disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] In this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0032] In the description of this disclosure, it should be understood that the terms "longitudinal", "length", "circumferential", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the subsystem or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.

[0033] Throughout the accompanying drawings, identical elements are represented by the same or similar reference numerals. Conventional structures or constructions have been omitted where they may cause confusion in understanding this disclosure. Furthermore, the shapes, dimensions, and positional relationships of the components in the drawings do not reflect actual size, scale, or actual positional relationships. Additionally, any reference numerals placed between parentheses in the claims should not be construed as limiting the claims.

[0034] Similarly, to simplify this disclosure and aid in understanding one or more of the various aspects of the disclosure, in the above description of exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. The use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refers to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present disclosure. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0035] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0036] This disclosure provides a method for multi-scale target detection in remote sensing images based on a cross-fusion detection head, comprising: extracting multiple sets of scale feature maps from the original image based on a feature extraction module; inputting the multiple sets of scale feature maps into a feature fusion module to obtain a fused feature map based on the multiple sets of scale feature maps; processing the fused feature map based on multiple branches in the cross-fusion detection head, and obtaining a target recognition result by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and the processing results output by different branches are different.

[0037] Figure 1 A flowchart illustrating a multi-scale target detection method for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure is shown.

[0038] like Figure 1 As shown, the remote sensing image multi-scale target detection method based on cross-fusion detection head in this embodiment includes operations S110 to S130.

[0039] In operation S110, multiple sets of scale feature maps are extracted from the original image based on the feature extraction module.

[0040] In some embodiments, a feature extraction module extracts multiple sets of scale feature maps from the original image. These feature maps contain image information at different scales and levels, such as edges, textures, and shapes.

[0041] In the specific implementation process, operation S110 may further include performing operations S1101 to S1102.

[0042] Figure 2 A schematic diagram illustrating the structure of a feature extraction module according to an embodiment of the present disclosure is shown. The following will be combined with... Figure 2 The operations S1101 to S1102 of the embodiments of this disclosure will be further described.

[0043] In operation S1101, features are extracted from the original image based on different extraction branches to obtain multiple sets of feature maps; among them, different networks are used for different extraction branches.

[0044] like Figure 2 As shown, the feature extraction module includes an attention mechanism branch (e.g., following the SwinTransformer architecture) and a convolutional branch (e.g., following the ResNet50 architecture). After obtaining the original image, it is simultaneously fed into both the attention mechanism branch and the convolutional branch of the feature extraction module. Features are extracted from the original image based on different extraction branches, resulting in multiple sets of feature maps {F2, F3, F4, F5}. These four sets of feature maps have scales of 1 / 32, 1 / 16, 1 / 8, and 1 / 4 of the original image, respectively, and have 2048, 1024, 512, and 256 channels, respectively.

[0045] In operation S1102, multiple sets of feature maps are transformed and fused based on different channels to obtain multiple sets of scale feature maps.

[0046] In some embodiments, in addition to extracting branches, the feature extraction module also includes a feature transformation and fusion unit, which is responsible for agreeing on the styles of two branches and interactively and continuously eliminating the semantic differences between them, thereby obtaining multiple sets of scale feature maps.

[0047] Specifically, the feature transformation and fusion unit comprises two structures in different directions. Channel 1 is responsible for unifying the feature maps extracted by the convolutional branch (i.e., ResNet50) to the output of the attention mechanism branch (SwinTransformer) at the previous scale, fusing the two, and then sending them to the next extraction unit in the attention mechanism branch. Channel 2 is responsible for unifying the feature maps extracted by the attention mechanism branch to the output of the convolutional branch at the next scale, fusing the two, and then sending them to the next extraction unit in the convolutional branch. This process is repeated to obtain the final four sets of scale feature maps to be output. Figure 2 It can be seen that in the process of operating S1102, channel one is reused 4 times, channel two is reused 3 times, and the final output is four scale feature maps from the attention mechanism branch.

[0048] In operation S120, multiple sets of scale feature maps are input to the feature fusion module to obtain a fused feature map based on the multiple sets of scale feature maps.

[0049] In some embodiments, the feature fusion module can be, for example, an enhanced feature pyramid region network. This module upsamples the scale feature map sequentially using bilinear interpolation and deconvolution to match the size of the feature map at the next scale. The upsampled feature map is then added to the feature map at the corresponding scale to obtain a new fused feature map. This process is repeated until fused feature maps corresponding to all scale feature maps are generated.

[0050] Figure 3 The schematic diagram illustrates the structure of a feature fusion module according to an embodiment of the present disclosure.

[0051] like Figure 3 As shown, in the specific implementation process, the feature fusion module processes the received multiple sets of scale feature maps {F2, F3, F4, F5} accordingly to obtain the fused feature map corresponding to that scale feature map. Specifically, as shown... Figure 3 As shown, convolution is performed on F2, outputting F2'. Bilinear interpolation is used to upsample F2 by a factor of two, and then added to F3 to obtain F3'. Deconvolution is used to upsample F3 by a factor of two, and then added to F4 to obtain F4'. Bilinear interpolation is used to upsample F4 by a factor of two, and then added to F5 to obtain F5'. Deconvolution is used to upsample F5 by a factor of two to obtain F6'. In this embodiment, bilinear interpolation and deconvolution are used for upsampling respectively. The formula for calculating the deconvolution output size is as follows:

[0052] out=(in-1)*stride-2*padding+kernel_size+output_padding.

[0053] Where out represents the output size and in represents the output size.

[0054] After obtaining the fused feature maps {F2', F3', F4', F5', F6'} corresponding to all scale feature maps, the fused feature maps are output to the cross-fusion detection head.

[0055] In operation S130, the fused feature map is processed based on multiple branches in the cross-fusion detection head. The target recognition result is obtained by cross-fusion of the processing results of each branch. The cross-fusion detection head includes multiple branches, and the processing results output by different branches are different.

[0056] In some embodiments, the cross-fusion detection head consists of a cross-interacting region candidate structure and an information-sharing hybrid rotated bounding box generation structure. The region candidate structure is primarily used to extract a series of candidate regions that may contain the target, along with the probability of the target's presence in each candidate region, from the input fused feature map. This effectively reduces the number of regions that need to be processed, providing a foundation for subsequent fine-grained target object detection and thus improving the detection efficiency. The hybrid rotated bounding box generation structure is used to more precisely adjust the candidate regions based on information between different feature maps, resulting in more accurate target detection results (i.e., target detection boxes and classification results).

[0057] In some embodiments, operation S130 may further include operations S131 to S132.

[0058] In operation S131, the fused feature map is input into the region candidate structure. Different first branches in the region candidate structure process the fused feature map, and the processing results of each branch are fused to obtain the first information.

[0059] In operation S132, the first information is input into the hybrid rotating frame generation structure. Different second branches in the hybrid rotating frame generation structure process the first information and cross-merge the processing results of each second branch to obtain the target recognition result.

[0060] Figure 4 A schematic diagram illustrating the structure of a region candidate structure according to an embodiment of the present disclosure is shown. The following will be combined with... Figure 4 The operation S131 will be described in further detail.

[0061] like Figure 4 As shown, in this embodiment of the present disclosure, the region candidate structure includes a first branch module and a first branch fusion module, wherein the first branch module further includes a first classification branch unit and a first regression branch unit.

[0062] In some embodiments, operation S131 may further include operations S1311 to S1312.

[0063] In operation S1311, the fused feature map is input to the first classification branch unit and the first regression branch unit respectively. The first classification branch unit enhances the regression details of pixels near the target object and suppresses the regression of non-target regions, while the first regression branch captures the scale information of the target object.

[0064] In the specific implementation process, the fused feature maps {F2', F3', F4', F5', F6'} output by the feature fusion module are input to the first classification branch unit and the first regression branch unit respectively after being convolved by 3×3. The first classification branch unit and the first regression branch unit process the input fused feature maps simultaneously.

[0065] The first classification branch is mainly used to predict the category of each candidate region. It extracts feature maps from the input image, generates a series of candidate regions on these feature maps, and further processes the feature maps of these candidate regions to enhance feature information related to the target object. In the first classification branch unit, regions surrounding the target object in the fused feature map remain active, while non-target regions gradually approach zero, effectively enhancing pixels and regression details near the target object and suppressing regression information in the target region. Specifically, the first classification branch unit uses three 1×1 convolutions to represent the foreground scores of the three predicted anchor points in the H and W directions.

[0066] The first regression branch is primarily used to predict the location information of each candidate region, i.e., the bounding box of the target object. It processes the input fused feature map to extract feature information related to the target object's location and uses this information to adjust the position of each candidate region, improving the accuracy of the bounding box information and enabling the bounding box to more accurately enclose the target object. The first regression branch also extracts feature maps from the input image and generates a series of candidate regions on these maps. Further processing is performed on these candidate regions to extract feature information related to the target object's location. The first regression branch can capture the scale information of the target object, where different objects exhibit different scale distributions. Specifically, 12 channels are obtained through 12 1×1 convolutions, i.e., 3 anchor points, and each anchor point requires 4 parameters to adjust its position.

[0067] In operation S1312, the regression details and the scale information are input to the first branch fusion module, and the first information is determined based on the first branch fusion module. The first information includes the target bounding box information and the probability of the target object existing.

[0068] In the specific implementation process, the processing results output by the first classification branch and the first regression branch are input to the first branch fusion module. The first branch fusion module performs convolution on the processing results output by the first classification branch and the first regression branch to unify the number of channels in the processing results of each branch. The unified branch processing results are then fused with the initial channel information to obtain the first information. Specifically, the output size of each branch is changed by a 3×3 convolution with padding=1, the number of channels of the features is modified by a 1×1 convolution, and the ReLU activation function is applied to the convolutional feature map. The unified feature map is added to and fused with the initial signal information, and the first information is determined based on the score of the fusion result. The top N results with the highest scores after fusion can be determined as the target box information and the probability of the target object's existence. N is a positive integer and can be determined according to the actual situation. In this embodiment, N=2000.

[0069] After determining the first information based on the region candidate structure, the first information is input into the hybrid rotation bounding box generation structure. The target bounding boxes in the first information are then fine-tuned through hybrid rotation to obtain the final target recognition result. On one hand, hybrid rotation fine-tuning can reduce interference and avoid overlapping detection boxes, thereby capturing the feature information of the target object at different angles and improving detection accuracy. On the other hand, by fine-tuning the target bounding boxes through hybrid rotation, the adaptability to complex scenes can be effectively improved, enabling the target detection method to accurately detect objects in any direction and improving target detection accuracy.

[0070] Figure 5 A schematic diagram illustrating the structure of the hybrid rotating frame generation structure according to an embodiment of the present disclosure is shown. The following will be combined with... Figure 5 The operation S132 will be described in further detail.

[0071] like Figure 5 As shown, the hybrid rotating box generation structure provided in this embodiment includes a shared fully convolutional module, a second branch module, and a second branch fusion module, wherein the second branch module includes a second classification branch unit and a second regression branch unit.

[0072] In some embodiments, operation S132 may further include operations S1321 to S1323.

[0073] In operation S1321, the first information output by the region candidate structure is input into the shared full convolution module, which performs a shared convolution on the first information, and then inputs the convolved first information into the second regression branch and the second classification branch respectively.

[0074] In some embodiments, the first information output by the region candidate structure is input to a shared fully convolutional module, which performs a shared convolution on the first information. The convolved first information is then input to the second regression branch and the second classification branch, respectively. The shared convolutional module comprises two 3×3 shared convolutional units.

[0075] In operation S1322, the second regression branch unit and the second classification branch unit process the input first information to obtain temporary location information and temporary category information.

[0076] In some embodiments, the second regression branch unit consists of two convolutions. After fusion, the temporary location information obtained by the second regression branch is output to the second branch fusion module through a fully connected layer. The second classification branch unit consists of a convolution and a fully connected layer. The first information input to the second classification branch unit from the shared convolution module is processed by the second classification branch unit to obtain temporary category information. This temporary category information is then output to the second branch fusion module after passing through a fully connected layer.

[0077] In operation S1323, the second branch fusion module cross-fused the temporary location information, temporary category information, and the first information to determine the target recognition result, wherein the target recognition result includes target category information and target location information.

[0078] In some embodiments, the branch fusion module of the hybrid rotated bounding box generation structure unifies the number of channels in the second regression branch and the second classification branch through a 1×1 convolution, and fuses the temporary category information and temporary location information of the target obtained through the classification branch and the regression branch, respectively. The temporary category information and temporary location information after channel transformation are interleaved with and added to the original branch information to obtain the target recognition result, wherein the target recognition result includes target category information and target location information. After obtaining the target recognition result, the target category information and target location information in the target recognition result can be mapped onto the original image, and the target recognition result is displayed in the form of detection boxes and detection results.

[0079] The cross-fusion detection head used in the multi-scale target detection method for remote sensing images provided in this disclosure effectively utilizes the location information implicit in the classification branch and the classification information implicit in the regression branch. By cross-fusioning the information from the classification and regression branches, the information barriers between them are broken down, enabling effective utilization of the implicit information in different branches. This enriches semantic information and enhances feature representation, thereby improving the accuracy of target category and target location, and effectively increasing target detection precision. Furthermore, the information from the regression and classification branches can mutually verify and complement each other during the fusion process, ensuring accurate target detection even in complex scenes and improving the robustness of the target detection method.

[0080] Based on the aforementioned method for multi-scale target detection in remote sensing images using a cross-fusion detection head, this disclosure also provides a device for multi-scale target detection in remote sensing images using a cross-fusion detection head. The following will combine... Figure 6 The device is described in detail.

[0081] Figure 6 A schematic block diagram of a multi-scale target detection device for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure is shown.

[0082] like Figure 6 As shown, the remote sensing image multi-scale target detection device 600 based on cross-fusion detection head in this embodiment includes a feature extraction module 610, a feature fusion module 620, and a cross-fusion detection head 630.

[0083] The feature extraction module 610 is used to extract multiple sets of scale feature maps from the original image. In one embodiment, the feature extraction module 610 can be used to perform the operation S110 described above, which will not be repeated here.

[0084] The feature fusion module 620 is used to obtain a fused feature map based on multiple sets of scale feature maps. In one embodiment, the feature fusion module 620 can be used to perform the operation S120 described above, which will not be repeated here.

[0085] The cross-fusion detection head 630 is used to perform branching processing on the fused feature map, and to obtain the target recognition result by cross-fusion of the processing results of each branch. The cross-fusion detection head includes multiple branches, and the processing results output by different branches are different. In one embodiment, the cross-fusion detection head 630 can be used to perform the operation S60 described above, which will not be repeated here.

[0086] According to embodiments of this disclosure, any plurality of modules among the feature extraction module 610, feature fusion module 620, and cross-fusion detection head 630 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the feature extraction module 610, feature fusion module 620, and cross-fusion detection head 630 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the feature extraction module 610, feature fusion module 620, and cross-fusion detection head 630 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0087] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a multi-scale target detection method for remote sensing images based on a cross-fusion detection head, according to embodiments of the present disclosure.

[0088] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0089] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0090] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0091] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this disclosure, those skilled in the art can make various substitutions and modifications, all of which should be included within the protection scope of this disclosure.

Claims

1. A method for multi-scale target detection in remote sensing images based on a cross-fusion detection head, characterized in that, include: The feature extraction module extracts multiple sets of scale feature maps from the original image; The multiple sets of scale feature maps are input into the feature fusion module to obtain a fused feature map based on the multiple sets of scale feature maps; The fused feature map is processed based on multiple branches in the cross-fusion detection head. The target recognition result is obtained by cross-fusion of the processing results of each branch. The cross-fusion detection head includes multiple branches, and the processing results output by different branches are different. The cross-fusion detection head includes a region candidate structure and a hybrid rotated bounding box generation structure. The fused feature map is processed based on multiple branches in the cross-fusion detection head, and the target recognition result is obtained by cross-fusion of the processing results of each branch, including: The fused feature map is input into the region candidate structure, and different first branches in the region candidate structure process the fused feature map and fuse the processing results of each branch to obtain first information; the region candidate structure includes a first branch module and a first branch fusion module, and the first branch module includes a first classification branch unit and a first regression branch unit; The first information is input into the hybrid rotating frame generation structure, and different second branches in the hybrid rotating frame generation structure process the first information and cross-merge the processing results of each second branch to obtain the target recognition result. The step of inputting the fused feature map into the region candidate structure, processing the fused feature map by different first branches in the region candidate structure, and fusing the processing results of each branch to obtain first information includes: inputting the fused feature map into a first classification branch unit and a first regression branch unit respectively; the first classification branch unit enhances the regression details of pixels near the target object and suppresses the regression of non-target regions; the first regression branch captures the scale information of the target object; inputting the regression details and the scale information into a first branch fusion module; and determining the first information based on the first branch fusion module. The first information includes target bounding box information and the probability of the target object existing.

2. The method for multi-scale target detection in remote sensing images according to claim 1, characterized in that, The step of inputting the regression details and the scale information into the first branch fusion module, and determining the first information based on the first branch fusion module, includes: Convolution is performed on the regression details and the scale information to unify the number of channels for the regression details and the scale information; The unified regression details and scale information are fused with the initial channel information to obtain the first information.

3. The method for multi-scale target detection in remote sensing images according to claim 1, characterized in that, The hybrid rotating box generation structure includes a shared fully convolutional module, a second branch module, and a second branch fusion module, wherein the second branch module includes a second classification branch unit and a second regression branch unit; The first information is input into the hybrid rotating frame generation structure, and different second branches in the hybrid rotating frame generation structure process the first information and cross-fuse the processing results of each second branch to obtain the target recognition result, including: The first information output by the region candidate structure is input into the shared full convolution module, and the shared full convolution module performs a shared convolution on the first information. The first information after convolution is then input into the second regression branch and the second classification branch respectively. The second regression branch unit and the second classification branch unit process the input first information respectively to obtain temporary location information and temporary category information; The second branch fusion module cross-fused the temporary location information, temporary category information, and first information to determine the target recognition result, wherein the target recognition result includes target category information and target location information.

4. The method for multi-scale target detection in remote sensing images according to claim 1, characterized in that, The feature extraction module extracts multiple sets of feature maps from the original image, including: Feature extraction is performed on the original image based on different extraction branches to obtain multiple sets of feature maps; different networks are used in different extraction branches. The multiple sets of feature maps are transformed and fused based on different channels to obtain multiple sets of scale feature maps.

5. The method for multi-scale target detection in remote sensing images according to claim 1, characterized in that, The step of inputting the multiple sets of scale feature maps into the feature fusion module to obtain fusion information includes: Perform the following operations on each scale feature map: Upsample the current scale feature map using bilinear interpolation and deconvolution to match the size of the current scale feature map with that of the next scale feature map. The upsampled feature map is added to the feature map at the corresponding scale to obtain the fused feature map corresponding to the current scale feature map; Repeat the above process until the fused feature map corresponding to all scale feature maps is obtained.

6. The method for multi-scale target detection in remote sensing images according to claim 1, characterized in that, Also includes: The target recognition results are mapped onto the original image, and the target recognition results are displayed in the form of detection boxes and detection results.

7. A multi-scale target detection device for remote sensing images based on a cross-fusion detection head, characterized in that, include: The feature extraction module is used to extract multiple sets of scale feature maps from the original image; The feature fusion module is used to obtain a fused feature map based on the multiple sets of scale feature maps; The cross-fusion detection head is used to perform branching processing on the fused feature map and to obtain the target recognition result by cross-fusion of the processing results of each branch. The cross-fusion detection head includes multiple branches, and the processing results output by different branches are different. The cross-fusion detection head includes a region candidate structure and a hybrid rotated bounding box generation structure. The fused feature map is processed based on multiple branches in the cross-fusion detection head, and the target recognition result is obtained by cross-fusion of the processing results of each branch, including: The fused feature map is input into the region candidate structure, and different first branches in the region candidate structure process the fused feature map and fuse the processing results of each branch to obtain first information; the region candidate structure includes a first branch module and a first branch fusion module, and the first branch module includes a first classification branch unit and a first regression branch unit; The first information is input into the hybrid rotating frame generation structure, and different second branches in the hybrid rotating frame generation structure process the first information and cross-merge the processing results of each second branch to obtain the target recognition result. The step of inputting the fused feature map into the region candidate structure, processing the fused feature map by different first branches in the region candidate structure, and fusing the processing results of each branch to obtain first information includes: inputting the fused feature map into a first classification branch unit and a first regression branch unit respectively; the first classification branch unit enhances the regression details of pixels near the target object and suppresses the regression of non-target regions; the first regression branch captures the scale information of the target object; inputting the regression details and the scale information into a first branch fusion module; and determining the first information based on the first branch fusion module. The first information includes target bounding box information and the probability of the target object existing.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

Images