Remote sensing image multi-scale target detection method, device and equipment based on cross fusion detection head

Abstract

The invention provides a remote sensing image multi-scale target detection method, device and equipment based on a cross fusion detection head, which can be applied to the technical field of images and the technical field of target detection, and comprises the following steps: extracting multiple groups of scale feature maps from an original image based on a feature extraction module; inputting the multiple groups of scale feature maps into a feature fusion module to obtain a fused feature map based on the multiple groups of scale feature maps; the fusion feature map is processed based on a plurality of branches in a cross fusion detection head, cross fusion is carried out on the processing result of each branch, a target recognition result is obtained, the cross fusion detection head comprises a plurality of branches, and the processing results output by different branches are different. The information in the classification branch and the regression branch is subjected to cross fusion through the cross fusion detection head, the information barrier between the classification branch and the regression branch is broken, the semantic information is enriched, the feature representation is enhanced, and the target detection precision is effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of image technology and target detection technology, and more specifically, to a method, device and equipment for multi-scale target detection in remote sensing images based on a cross-fusion detection head. Background Art

[0002] Remote sensing target monitoring is of great significance in the fields of information collection, distress rescue, real-time target tracking, etc. Traditional remote sensing image target detection methods rely on artificially designed features and are difficult to adapt to the interference of complex backgrounds and changing atmospheric environments.

[0003] In recent years, with the development of high-resolution optical satellite imaging technology and artificial intelligence, target detection, as a classic research direction in the field of image processing, has also faced new challenges and breakthroughs. Some scholars have applied deep learning to the target detection task of remote sensing images. Most of the targets in remote sensing images have the characteristics of diverse scales and complex backgrounds, which brings great difficulties to accurate detection. The detection head is the key module connecting feature extraction and the final detection results. Its design and optimization directly affect the performance of the target detection network, including the accuracy of detection and adaptability to different scales and complex backgrounds.

[0004] In the existing two-stage target detection network, the detection head usually consists of two branches: the classification branch and the regression branch. These branches independently provide target classification and location information. In the classification branch, the area around the target in the feature map remains active, and the non-target area gradually approaches zero. In the regression branch, the scale information of the captured object is also included, and different objects show different scale distributions. In the existing target detection network, the information of only one branch is simply used, which may lead to insufficient utilization of the location information of the classification branch and the classification information of the regression branch. As a result, when performing multi-scale target detection in remote sensing images, problems such as information loss and low detection accuracy caused by inaccurate recognition may occur. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The present disclosure provides a remote sensing image multi-scale target detection method, device and electronic equipment based on a cross-fusion detection head, which are used to at least partially solve one of the above-mentioned technical problems.

[0007] (II) Technical solution

[0008] According to a first aspect of the present disclosure, a multi-scale target detection method for remote sensing images based on a cross-fusion detection head is provided, comprising: extracting multiple groups of scale feature maps from the original image based on a feature extraction module; inputting the multiple groups of scale feature maps into a feature fusion module to obtain a fused feature map based on the multiple groups of scale feature maps; processing the fused feature map based on multiple branches in the cross-fusion detection head, and obtaining a target recognition result by cross-fusing the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and different branches output different processing results.

[0009] According to an embodiment of the present disclosure, a cross-fusion detection head includes a region candidate structure and a hybrid rotation frame generation structure. The fusion information is processed based on the branches in the cross-fusion detection head, and a target recognition result is obtained by fusing the processing results of each branch, including: inputting the fused feature map into the region candidate structure, processing the fused feature map by different first branches in the region candidate structure and fusing the processing results of each branch to obtain first information; inputting the first information into the hybrid rotation frame generation structure, processing the first information by different second branches in the hybrid rotation frame generation structure and cross-fusing the processing results of each second branch to obtain a target recognition result.

[0010] According to an embodiment of the present disclosure, a region candidate structure includes a first branch module and a first branch fusion module, the first branch module includes a first classification branch unit and a first regression branch unit; the fused feature map is input into the region candidate structure, and different first branches in the region candidate structure process the fused feature map and fuse the processing results of each branch to obtain first information, including: inputting the fused feature map into the first classification branch unit and the first regression branch unit respectively, the first classification branch unit enhances the regression details of pixels near the target object and suppresses the regression of non-target areas, and the first regression branch captures the scale information of the target object; the regression details and the scale information are input into the first branch fusion module, and the first information is determined based on the first branch fusion module, wherein the first information includes target frame information and the probability of the existence of the target object.

[0011] According to an embodiment of the present disclosure, the regression detail and scale information are input into a first branch fusion module, and the first information is determined based on the first branch fusion module, including: convolving the regression detail information and the scale information to unify the number of channels of the regression detail information and the scale information; fusing the unified regression detail information and scale information with the initial channel information to obtain the first information.

[0012] According to an embodiment of the present disclosure, a hybrid rotation frame generation structure includes a shared full convolution module, a second branch module and a second branch fusion module, wherein the second branch module includes a second classification branch unit and a second regression branch unit; the first information is input into the hybrid rotation frame generation structure, and different second branches in the hybrid rotation frame generation structure process the first information and cross-fuse the processing results of each second branch to obtain a target recognition result, including: inputting the first information output by the region candidate structure into a shared full convolution module, and the shared full convolution module performs shared convolution on the first information, and inputting the convolved first information into the second regression branch and the second classification branch respectively; the second regression branch unit and the second classification branch unit process the input first information respectively to obtain temporary position information and temporary category information; the second branch fusion module cross-fuses the temporary position information, the temporary category information and the first information to determine the target recognition result, wherein the target recognition result includes target category information and target position information.

[0013] According to an embodiment of the present disclosure, multiple groups of feature maps are extracted from the original image based on a feature extraction module, including: extracting features from the original image based on different extraction branches to obtain multiple groups of feature maps; wherein different extraction branches use different networks; and converting and fusing the multiple groups of feature maps based on different channels to obtain multiple groups of scale feature maps.

[0014] According to an embodiment of the present disclosure, multiple groups of scale feature maps are input into a feature fusion module to obtain fusion information, including: performing the following operations on each scale feature map: upsampling the current scale feature map using bilinear difference and deconvolution to match the size of the current scale feature map with the size of the next scale feature map; adding the upsampled feature map to the feature map of the corresponding scale to obtain a fused feature map corresponding to the current scale feature map; repeating the above process until the fused feature maps corresponding to all scale feature maps are obtained.

[0015] According to an embodiment of the present disclosure, it also includes: mapping the target recognition result to the original image, and displaying the target recognition result in the form of a detection frame and a detection result.

[0016] According to a second aspect of the present disclosure, a multi-scale target detection device for remote sensing images based on a cross-fusion detection head is provided, including: a feature extraction module, used to extract multiple groups of scale feature maps from the original image; a feature fusion module, used to obtain a fused feature map based on the multiple groups of scale feature maps; a cross-fusion detection head, used to perform branch processing on the fused feature map, and obtain a target recognition result by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and different branches output different processing results.

[0017] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0018] (III) Beneficial effects

[0019] The remote sensing image multi-scale target detection method, device and electronic device based on the cross-fusion detection head provided by the present disclosure have at least the following beneficial effects:

[0020] The cross-fusion detection head is used to effectively break the information barrier between the classification branch and the regression branch in the detection head. By cross-summing the output of the regression branch and the output of the classification branch, more hidden information can be obtained. The classification branch can effectively enhance the regression details of pixels near the target and suppress the regression of non-target areas, effectively improving the accuracy of remote sensing target detection. In addition, the generated data is debugged using randomly generated proposed candidate boxes to overlay and refine the prediction of changes, thereby further improving the accuracy of remote sensing target detection and the precision of remote sensing image target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a multi-scale target detection method for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 2 A schematic diagram schematically shows the structure of a feature extraction module according to an embodiment of the present disclosure;

[0024] Figure 3 The structural diagram of the feature fusion module according to the embodiment of the present disclosure is schematically shown;

[0025] Figure 4 A schematic diagram schematically shows a structural diagram of a region candidate structure according to an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram schematically shows a structure of a hybrid rotation frame generation structure according to an embodiment of the present disclosure;

[0027] Figure 6 The structure block diagram of the multi-scale target detection device of remote sensing images based on the cross-fusion detection head according to the embodiment of the present disclosure is schematically shown;

[0028] Figure 7A block diagram of an electronic device suitable for implementing a multi-scale target detection method for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] In the present disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or can communicate with each other; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0032] In the description of the present disclosure, it is necessary to understand that the terms "longitudinal", "length", "circumferential", "front", "rear", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the subsystem or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0033] Throughout the drawings, the same elements are represented by the same or similar reference numerals. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present disclosure. The shapes, sizes, and positional relationships of the components in the drawings do not reflect the actual size, proportion, and actual positional relationship. In addition, in the claims, any reference symbol between brackets shall not be construed as a limitation to the claims.

[0034] Similarly, in order to simplify the present disclosure and help understand one or more of the various disclosed aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. The description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0035] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present disclosure, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0036] The disclosed embodiment provides a multi-scale target detection method for remote sensing images based on a cross-fusion detection head, comprising: extracting multiple groups of scale feature maps from the original image based on a feature extraction module; inputting the multiple groups of scale feature maps into a feature fusion module to obtain a fused feature map based on the multiple groups of scale feature maps; processing the fused feature map based on multiple branches in the cross-fusion detection head, and obtaining a target recognition result by cross-fusing the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and different branches output different processing results.

[0037] Figure 1 The flowchart of the multi-scale target detection method of remote sensing images based on the cross-fusion detection head according to the embodiment of the present disclosure is schematically shown.

[0038] like Figure 1 As shown, the multi-scale target detection method for remote sensing images based on a cross-fusion detection head of this embodiment includes operations S110 to S130.

[0039] In operation S110 , a plurality of groups of scale feature maps are extracted from the original image based on a feature extraction module.

[0040] In some embodiments, multiple groups of scale feature maps are extracted from the original image based on the feature extraction module. These feature maps contain image information of different scales and levels, such as edges, textures, shapes, etc.

[0041] In a specific implementation process, operation S110 may further include operation S1101 to operation S1102.

[0042] Figure 2 The following is a schematic diagram of the structure of the feature extraction module according to the embodiment of the present disclosure. Figure 2 Operation S1101 to operation S1102 of the embodiment of the present disclosure are further explained.

[0043] In operation S1101, feature extraction is performed on the original image based on different extraction branches to obtain multiple groups of feature maps; wherein different extraction branches use different networks.

[0044] like Figure 2 As shown in the figure, the feature extraction module includes an attention mechanism branch (for example, it can be constructed according to SwinTransformer) and a convolution branch (for example, it can be constructed according to ResNet50). After obtaining the original image, the original image is simultaneously sent to the attention mechanism branch and the convolution branch of the feature extraction module, and the original image is subjected to feature extraction based on different extraction branches to obtain multiple sets of feature maps {F2, F3, F4, F5}. Among them, the scales of these four sets of feature maps are 1 / 32, 1 / 16, 1 / 8, and 1 / 4 of the original image, respectively, and the number of channels is 2048, 1024, 512, and 256, respectively.

[0045] In operation S1102, multiple groups of feature maps are transformed and fused based on different channels to obtain multiple groups of scale feature maps.

[0046] In some embodiments, in addition to the extraction branch, the feature extraction module also includes a feature conversion fusion unit, which is responsible for agreeing on the styles of the two branches and continuously eliminating the semantic differences between them in an interactive manner, thereby obtaining multiple groups of scale feature maps.

[0047] Specifically, the feature conversion fusion unit includes two structures in different directions. Channel one is responsible for unifying the feature map extracted by the convolution branch (i.e., ResNet50) to the output of the previous scale of the attention mechanism branch (SwinTransformer), and fusing the two to the next layer of extraction unit in the attention mechanism branch. Channel two is responsible for unifying the feature map extracted by the attention mechanism branch to the output of the next scale of the convolution branch, and fusing the two to the next layer of extraction unit in the convolution branch. Repeat the above steps to obtain the four sets of scale feature maps to be output. Figure 2 It can be seen that in the process of operation S1102, channel 1 is reused 4 times, channel 2 is reused 3 times, and the final output is four scale feature maps from the attention mechanism branch.

[0048] In operation S120, the multiple groups of scale feature maps are input into a feature fusion module to obtain a fused feature map based on the multiple groups of scale feature maps.

[0049] In some embodiments, the feature fusion module may be, for example, an enhanced feature pyramid region network. The module upsamples the scale feature map by sequentially using bilinear interpolation and deconvolution to match the size of the feature map of the next scale. The upsampled feature map is added to the feature map of the corresponding scale to obtain a new fused feature map, and the above process is repeated until the fused feature maps corresponding to all scale feature maps are generated.

[0050] Figure 3 The structural diagram of the feature fusion module according to the embodiment of the present disclosure is schematically shown.

[0051] like Figure 3 As shown, in the specific implementation process, the feature fusion module processes the received multiple groups of scale feature maps {F2, F3, F4, F5} respectively to obtain the fusion feature map corresponding to the scale feature map. Specifically, as Figure 3 As shown, convolution is performed on F2, and F2 is output as F2'. F2 is upsampled twice using bilinear interpolation and added to F3 to obtain F3'. F3 is upsampled twice using deconvolution and added to F4 to obtain F4'. F4 is upsampled twice using bilinear interpolation and added to F5 to obtain F5'. F5 is upsampled twice using deconvolution to obtain F6'. In the disclosed embodiment, bilinear interpolation and deconvolution are used for upsampling respectively. The deconvolution output size calculation formula is as follows:

[0052] out=(in-1)*stride-2*padding+kernel_size+output_padding.

[0053] Among them, out represents the output size and in represents the output size.

[0054] After obtaining the fused feature maps {F2', F3', F4', F5', F6'} corresponding to all scale feature maps, the fused feature maps are output to the cross-fusion detection head.

[0055] In operation S130, the fused feature map is processed based on multiple branches in the cross-fusion detection head, and the target recognition result is obtained by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and different branches output different processing results.

[0056] In some embodiments, the cross-fusion detection head is composed of a cross-interactive region candidate structure and an information-sharing hybrid rotation frame generation structure. The region candidate structure is mainly used to extract a series of candidate regions that may contain targets and the probability of the existence of targets in each candidate region from the input fusion feature map, effectively reducing the number of regions that need to be processed, providing a basis for subsequent fine detection of target objects, thereby improving the detection efficiency of target objects. The hybrid rotation frame generation structure is used to make more precise adjustments to the candidate regions based on the information between different feature maps to obtain more accurate target detection results (i.e., target detection frames and classification results).

[0057] In some embodiments, operation S130 may further include operations S131 to S132.

[0058] In operation S131, the fused feature map is input into the region candidate structure, and different first branches in the region candidate structure process the fused feature map and fuse the processing results of each branch to obtain the first information

[0059] In operation S132, the first information is input into the hybrid rotation frame generation structure, and different second branches in the hybrid rotation frame generation structure process the first information and cross-fuse the processing results of the second branches to obtain a target recognition result.

[0060] Figure 4 The schematic diagram of the structure of the region candidate structure according to the embodiment of the present disclosure is schematically shown. Figure 4 Operation S131 is described in further detail.

[0061] like Figure 4 As shown, the region candidate structure in the embodiment of the present disclosure includes a first branch module and a first branch fusion module, wherein the first branch module further includes a first classification branch unit and a first regression branch unit.

[0062] In some embodiments, operation S131 may further include operation S1311 to operation S1312.

[0063] In operation S1311, the fused feature map is input into the first classification branch unit and the first regression branch unit respectively, the first classification branch unit enhances the regression details of the pixels near the target object and suppresses the regression of the non-target area, and the first regression branch captures the scale information of the target object.

[0064] In the specific implementation process, the fused feature map {F2', F3', F4', F5', F6'} output by the feature fusion module is respectively input to the first classification branch unit and the first regression branch unit after 3×3 convolution, and the first classification branch unit and the first regression branch unit process the input fused feature map simultaneously.

[0065] The first classification branch is mainly used to predict the category to which each candidate region belongs. The first classification branch extracts a feature map from the input image, generates a series of candidate regions on the feature map, and further processes the feature map of the candidate region to enhance the feature information related to the target object. In the first classification branch unit, the area around the target object in the fusion feature map remains active, and the non-target area gradually approaches 0, thereby effectively enhancing the pixels and regression details near the target object and suppressing the regression information of the target area. Specifically, the first classification branch unit represents the foreground scores of the three anchor points predicted in the H and W directions through three 1×1 convolutions.

[0066] The first regression branch is mainly used to predict the position information of each candidate area, that is, the bounding box of the target object. The first regression branch processes the input fusion feature map to extract feature information related to the position of the target object, and uses this information to adjust the position of each candidate area to improve the accuracy of the target box information so that the target box can more accurately surround the target object. The first regression branch also extracts a feature map from the input image and generates a series of candidate areas on the feature map. The candidate areas are further processed to extract feature information related to the position of the target object. The first regression branch can capture the scale information of the target object, where different objects show different scale distributions. Specifically, 12 channels are obtained through 12 1×1 convolutions, that is, 3 anchor points and each anchor point requires 4 parameters to adjust its position.

[0067] In operation S1312, the regression details and the scale information are input into a first branch fusion module, and first information is determined based on the first branch fusion module, where the first information includes target frame information and a probability of existence of the target object.

[0068] In the specific implementation process, the processing results output by the first classification branch and the first regression branch are input into the first branch fusion module, and the first branch fusion module convolves the processing results output by the first classification branch and the first regression branch to unify the number of channels of the processing results of each branch, and fuses the unified branch processing results with the initial channel information to obtain the first information. Specifically, the output size of each branch is changed by a 3×3 convolution with padding=1, the number of channels of the feature is modified by a 1×1 convolution, and the ReLU activation function is applied to the convolution feature map, the unified feature map is added and fused with the initial signal information, and the first information is determined according to the score of the fusion result. Among them, the top N results with the highest scores after fusion can be determined as the target frame information and the probability of the existence of the target object, N is a positive integer, which can be determined according to the actual situation. In the embodiment disclosed in the present invention, N=2000 is set.

[0069] After determining the first information based on the region candidate structure, the first information is input into the hybrid rotation frame generation structure, and the target frame in the first information is subjected to hybrid rotation fine-tuning to obtain the final target recognition result. On the one hand, hybrid rotation fine-tuning can reduce Beijing interference and avoid detection frame overlap to capture the feature information of the target object at different angles. Improve detection accuracy. On the other hand, hybrid rotation fine-tuning of the target frame can effectively improve the adaptability to complex scenes, so that the target detection method can accurately detect objects in any direction and improve the accuracy of target detection.

[0070] Figure 5 The structure diagram of the hybrid rotation frame generation structure according to the embodiment of the present disclosure is schematically shown. Figure 5 Operation S132 is described in further detail.

[0071] like Figure 5 As shown, the hybrid rotation frame generation structure provided by the embodiment of the present disclosure includes a shared full convolution module, a second branch module and a second branch fusion module, wherein the second branch module includes a second classification branch unit and a second regression branch unit.

[0072] In some embodiments, operation S132 may further include operations S1321 to S1323.

[0073] In operation S1321, the first information output by the region candidate structure is input into a shared full convolution module, the shared full convolution module performs shared convolution on the first information, and the convolved first information is respectively input into a second regression branch and a second classification branch.

[0074] In some embodiments, the first information output by the region candidate structure is input into a shared full convolution module, the shared full convolution module performs shared convolution on the first information, and the convolved first information is input into the second regression branch and the second classification branch, respectively. The shared convolution module includes two 3×3 shared convolution units.

[0075] In operation S1322, the second regression branch unit and the second classification branch unit process the input first information respectively to obtain temporary position information and temporary category information.

[0076] In some embodiments, the second regression branch unit is composed of two convolutions, and the fused second regression branch outputs the obtained temporary position information to the second branch fusion module through a fully connected layer. The second classification branch unit is composed of a convolution and a fully connected layer, and the first information input to the second classification branch unit by the shared convolution module is processed by the second classification branch unit to obtain temporary category information, and the temporary category information is output to the second branch fusion module after a fully connected layer.

[0077] In operation S1323, the second branch fusion module cross-fuses the temporary position information, the temporary category information, and the first information to determine a target recognition result, wherein the target recognition result includes target category information and target position information.

[0078] In some embodiments, the branch fusion module of the hybrid rotating frame generation structure unifies the number of channels of the second regression branch and the second classification branch through 1×1 convolution, and fuses the temporary category information and temporary position information of the target obtained through the classification branch and the regression branch respectively. The temporary category information and temporary position information after channel conversion are cross-linked and added with the original branch information to obtain the target recognition result, wherein the target recognition result includes the target category information and the target position information. After obtaining the target recognition result, the target category information and the target position information in the target recognition result can be mapped to the original image, and the target recognition result can be displayed in the form of a detection frame and a detection result.

[0079] The cross-fusion detection head used in the multi-scale target detection method for remote sensing images provided by the embodiment of the present disclosure effectively utilizes the position information implicit in the classification branch and the classification information implicit in the regression branch. By cross-fusion of the information in the classification branch and the regression branch, the information barrier between the classification branch and the regression branch is broken, and the implicit information in different branches is effectively utilized. The semantic information is enriched while the feature representation is enhanced, thereby improving the accuracy of the target category and the target position positioning, and effectively improving the target detection accuracy. In addition, the regression branch and the classification branch information can also verify and complement each other during the fusion process, ensuring that accurate target detection can be achieved even in complex scenarios, and improving the robustness of the target detection method.

[0080] Based on the above-mentioned remote sensing image multi-scale target detection method based on cross-fusion detection head, the present disclosure also provides a remote sensing image multi-scale target detection device based on cross-fusion detection head. Figure 6 The device is described in detail.

[0081] Figure 6 The structural block diagram of a multi-scale target detection device for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 6 As shown, the remote sensing image multi-scale target detection device 600 based on the cross-fusion detection head of this embodiment includes a feature extraction module 610, a feature fusion module 620 and a cross-fusion detection head 630.

[0083] The feature extraction module 610 is used to extract multiple groups of scale feature maps from the original image. In one embodiment, the feature extraction module 610 can be used to perform the operation S110 described above, which will not be described in detail here.

[0084] The feature fusion module 620 is used to obtain a fused feature map based on multiple groups of scale feature maps. In one embodiment, the feature fusion module 620 can be used to perform the operation S120 described above, which will not be described in detail here.

[0085] The cross-fusion detection head 630 is used to perform branch processing on the fusion feature map, and obtain the target recognition result by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and the processing results output by different branches are different. In one embodiment, the cross-fusion detection head 630 can be used to perform the operation S60 described above, which will not be repeated here.

[0086] According to an embodiment of the present disclosure, any multiple modules of the feature extraction module 610, the feature fusion module 620 and the cross-fusion detection head 630 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the feature extraction module 610, the feature fusion module 620 and the cross-fusion detection head 630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the feature extraction module 610, the feature fusion module 620 and the cross-fusion detection head 630 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0087] Figure 7 A block diagram of an electronic device suitable for implementing a multi-scale target detection method for remote sensing images based on a cross-fusion detection head according to an embodiment of the present disclosure is schematically shown.

[0088] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0089] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0090] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0091] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all be included in the scope of protection of the present disclosure.

Claims

1. A multi-scale target detection method for remote sensing images based on a cross-fusion detection head, characterized in that: include: Extract multiple groups of scale feature maps from the original image based on the feature extraction module; Inputting the multiple groups of scale feature maps into a feature fusion module to obtain a fused feature map based on the multiple groups of scale feature maps; The fused feature map is processed based on multiple branches in a cross-fusion detection head, and the target recognition result is obtained by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and different branches output different processing results.

2. The remote sensing image multi-scale target detection method according to claim 1, characterized in that: The cross-fusion detection head includes a region candidate structure and a hybrid rotation frame generation structure. The branches in the cross-fusion detection head are used to process the fusion information, and the target recognition result is obtained by fusing the processing results of the branches, including: Inputting the fused feature map into the region candidate structure, processing the fused feature map by different first branches in the region candidate structure and fusing the processing results of each branch to obtain first information; The first information is input into a hybrid rotation frame generation structure, and different second branches in the hybrid rotation frame generation structure process the first information and cross-fuse the processing results of the second branches to obtain a target recognition result.

3. The remote sensing image multi-scale target detection method according to claim 2, characterized in that: The region candidate structure includes a first branch module and a first branch fusion module, and the first branch module includes a first classification branch unit and a first regression branch unit; The step of inputting the fused feature map into the region candidate structure, processing the fused feature map by different first branches in the region candidate structure and fusing the processing results of each branch to obtain first information includes: The fused feature map is input into a first classification branch unit and a first regression branch unit respectively, the first classification branch unit enhances the regression details of pixels near the target object and suppresses the regression of non-target areas, and the first regression branch captures the scale information of the target object; The regression details and the scale information are input into a first branch fusion module, and first information is determined based on the first branch fusion module, wherein the first information includes target frame information and a probability of existence of a target object.

4. The remote sensing image multi-scale target detection method according to claim 3, characterized in that: The step of inputting the regression details and the scale information into a first branch fusion module, and determining first information based on the first branch fusion module, comprises: Convolving the regression detail information and the scale information to unify the number of channels of the regression detail information and the scale information; The unified regression detail information and scale information are fused with the initial channel information to obtain first information.

5. The remote sensing image multi-scale target detection method according to claim 2, characterized in that: The hybrid rotation frame generation structure includes a shared full convolution module, a second branch module and a second branch fusion module, wherein the second branch module includes a second classification branch unit and a second regression branch unit; The step of inputting the first information into a hybrid rotation frame generation structure, processing the first information by different second branches in the hybrid rotation frame generation structure and cross-merging the processing results of the second branches to obtain a target recognition result includes: Inputting the first information output by the region candidate structure into the shared full convolution module, performing shared convolution on the first information by the shared full convolution module, and inputting the convolved first information into the second regression branch and the second classification branch respectively; The second regression branch unit and the second classification branch unit respectively process the input first information to obtain temporary position information and temporary category information; The second branch fusion module cross-fuses the temporary location information, the temporary category information and the first information to determine a target recognition result, wherein the target recognition result includes target category information and target location information.

6. The remote sensing image multi-scale target detection method according to claim 1, characterized in that: The extracting multiple groups of feature maps from the original image based on the feature extraction module includes: Based on different extraction branches, features of the original image are extracted to obtain multiple sets of feature maps; different extraction branches use different networks; The multiple groups of feature maps are transformed and fused based on different channels to obtain multiple groups of scale feature maps.

7. The remote sensing image multi-scale target detection method according to claim 1, characterized in that: The step of inputting the multiple groups of scale feature maps into a feature fusion module to obtain fusion information includes: Perform the following operations on each scale feature map: Use bilinear interpolation and deconvolution to upsample the current scale feature map so that the size of the current scale feature map matches that of the next scale feature map; Adding the upsampled feature map to the feature map of the corresponding scale to obtain a fused feature map corresponding to the feature map of the current scale; Repeat the above process until the fused feature maps corresponding to all scale feature maps are obtained.

8. The remote sensing image multi-scale target detection method according to claim 1, characterized in that: Also includes: The target recognition result is mapped to the original image, and the target recognition result is displayed in the form of a detection frame and a detection result.

9. A multi-scale target detection device for remote sensing images based on a cross-fusion detection head, characterized in that: include: A feature extraction module is used to extract multiple groups of scale feature maps from the original image; A feature fusion module, used for obtaining a fused feature map based on the multiple groups of scale feature maps; The cross-fusion detection head is used to perform branch processing on the fusion feature map, and obtain the target recognition result by cross-fusion of the processing results of each branch, wherein the cross-fusion detection head includes multiple branches, and different branches output different processing results.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images