Target detection method for performing multi-modal fusion according to adaptive weight

Abstract

The invention relates to a target detection method for performing multi-modal fusion according to adaptive weights. The method comprises the following steps: acquiring an infrared image and a visible light image shot by an infrared camera and a visible light camera in the same scene; inputting the infrared image and the visible light image into a pre-trained target detection model for target detection to obtain a target detection result; wherein the target detection model comprises a backbone network; the backbone network comprises an infrared image feature extraction module, a visible light image feature extraction module, an adaptive modal fusion weight calculation module and a cross-modal feature interaction module; and the adaptive modal fusion weight calculation module comprises a first independent modal weight calculation module, a second independent modal weight calculation module, a bias item calculation module and a softmax layer, so that the problem of inaccurate target detection caused by a mode of performing weighted summation by adopting a fixed weight in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a target detection method for performing multi-modal fusion according to adaptive weights. Background Art

[0002] Visible light images are suitable for use under normal lighting conditions because of their rich colors and detailed information, but their performance is relatively poor in dark or low-visibility conditions. In contrast, infrared images can provide stable target information in low-light or even dark environments, but because they only have a single color channel, their imaging quality is relatively simple and their information expression capabilities are limited. Therefore, fusing visible light and infrared images can use the advantages of both modalities to improve the accuracy of target detection.

[0003] At present, in the existing target detection models, it is generally implemented by weighted summation with fixed weights. However, since it ignores the dynamic changes in image quality, in practical applications, the quality of visible light and infrared images in different environments varies greatly, and fixed weights cannot flexibly respond to these changes, resulting in poor fusion effects in changing scenes, thus causing the problem of inaccurate target detection. Summary of the invention

[0004] 1. Technical issues to be resolved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a target detection method for multimodal fusion based on adaptive weights, which solves the technical problem of inaccurate target detection in the prior art.

[0005] (II) Technical solution In order to achieve the above object, the main technical solutions adopted by the present invention include: In a first aspect, an embodiment of the present invention provides a target detection method for performing multimodal fusion according to adaptive weights, comprising: obtaining an infrared image and a visible light image of the same scene captured by an infrared camera and a visible light camera; simultaneously inputting the infrared image and the visible light image into a pre-trained target detection model for target detection to obtain a target detection result; wherein the target detection model comprises a backbone network and a neck network; the backbone network comprises an infrared image feature extraction module, a visible light image feature extraction module, an adaptive modal fusion weight calculation module and a cross-modal feature interaction module; the infrared image feature extraction module and the visible light image feature extraction module both comprise P1 to P5 layers; the adaptive modal fusion weight calculation module comprises a first independent modal weight calculation module, a second independent modal weight calculation module, a bias item calculation module and a softmax layer, and the first independent The modal weight calculation module is used to determine the first preliminary weight value according to the feature map output by the P5 layer of the infrared image feature extraction module, and the second independent modal weight calculation module is used to determine the second preliminary weight value according to the feature map output by the P5 layer of the visible light image feature extraction module, and the bias item calculation module is used to determine the bias item according to the feature map output by the P3 layer of the infrared image feature extraction module and the feature map output by the P3 layer of the visible light image feature extraction module, and the softmax layer is used to determine the first modal fusion weight and the second modal fusion weight according to the first preliminary weight value, the second preliminary weight value and the bias item; the cross-modal feature interaction module is used to fuse the corresponding feature maps according to the first modal fusion weight and the second modal fusion weight to obtain a fused feature map; the neck network is used to output the target detection result according to the fused feature map.

[0006] In a possible embodiment, the first independent modality weight calculation module and the second independent modality weight calculation module have the same structure; the first independent modality weight calculation module and the second independent modality weight calculation module both include a first convolutional layer, a downsampling layer and a first multi-layer perception layer connected in sequence.

[0007] In one possible embodiment, the bias item calculation module includes a second convolutional layer, a third convolutional layer, a multiplication operation layer, a subtraction operation layer, a fourth convolutional layer and a second multi-layer perception layer, the second convolutional layer is respectively connected to the P3 layer, the multiplication operation layer and the subtraction operation layer of the infrared image feature extraction module, and the third convolutional layer is respectively connected to the P3 layer and the multiplication operation layer of the visible light image feature extraction module, and the multiplication operation layer is also connected to the subtraction operation layer, and the subtraction operation layer is also connected to the fourth convolutional layer, and the fourth convolutional layer is also connected to the second multi-layer perception layer.

[0008] In a possible embodiment, the cross-modal feature interaction module includes a first cross-modal feature interaction sub-module, a second cross-modal feature interaction sub-module and a third cross-modal feature interaction sub-module, the first cross-modal feature interaction sub-module is respectively connected to the softmax layer, the P3 layer of the infrared image feature extraction module and the P3 layer of the visible light image feature extraction module, and the second cross-modal feature interaction sub-module is respectively connected to the softmax layer, the P4 layer of the infrared image feature extraction module and the P4 layer of the visible light image feature extraction module, and the third cross-modal feature interaction sub-module is respectively connected to the softmax layer, the P5 layer of the infrared image feature extraction module and the P5 layer of the visible light image feature extraction module.

[0009] In a possible embodiment, the first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule, and the third cross-modal feature interaction submodule have the same structure; The first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule and the third cross-modal feature interaction submodule all include a fourth convolutional layer, a fifth convolutional layer, a first addition operation layer, a Transformer layer, a sixth convolutional layer, a seventh convolutional layer, a GSConv layer, an eighth convolutional layer, a second addition operation layer, a connection layer and a ninth convolutional layer, the first addition operation layer is connected to the fourth convolutional layer, the fifth convolutional layer and the Transformer layer, respectively, and the Transformer layer is also connected to the sixth convolutional layer and the seventh convolutional layer, respectively, and the sixth convolutional layer is also connected to the second addition operation layer, and the seventh convolutional layer is also connected to the GSConv layer and the eighth convolutional layer, and the second addition operation layer is also connected to the GSConv layer and the connection layer, and the connection layer is also connected to the eighth convolutional layer and the ninth convolutional layer.

[0010] In one possible embodiment, the Transformer layer includes a position encoding layer, a third addition operation layer, a first normalization layer, a multi-head attention layer, a fourth addition operation layer, a second normalization layer, a third multi-layer perception layer and a fifth addition operation layer, the third addition operation layer is respectively connected to the position encoding layer, the first normalization layer and the fourth addition operation layer, and the first normalization layer is connected to the multi-head attention layer, and the multi-head attention layer is also connected to the fourth addition operation layer, and the fourth addition operation layer is respectively connected to the second normalization layer and the fifth addition operation layer, and the second normalization layer is also connected to the third multi-layer perception layer, and the third multi-layer perception layer is also connected to the fifth addition operation layer.

[0011] In a possible embodiment, the calculation expression of the first modality fusion weight is: ; In the formula, WIR represents the first modality fusion weight; soft max means soft max operation; w 1 represents the first preliminary weight value; w 2 represents the second preliminary weight value; b represents the bias term; And, the calculation expression of the second modality fusion weight is: ; In the formula, WrG Represents the second modality fusion weight.

[0012] In a second aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to execute the method described in the first aspect or any optional implementation of the first aspect.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the method described in the first aspect or any optional implementation method of the first aspect is performed.

[0014] In a fourth aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0015] (III) Beneficial effects The beneficial effects of the present invention are: The embodiment of the present application provides a target detection method for multimodal fusion based on adaptive weights, by obtaining infrared images and visible light images of the same scene taken by an infrared camera and a visible light camera, and then simultaneously inputting the infrared images and visible light images into a pre-trained target detection model for target detection, and obtaining target detection results. Among them, the target detection model includes a branch for extracting infrared image features from infrared images and another branch for extracting visible light image features from visible light images. For a given infrared image and visible light image, after feature extraction through a dual-stream network, the information of the dual modalities is integrated into three output streams of different sizes through a cross-modal feature interaction module at three different levels of the backbone network part of the target detection model. At the same time, the weights calculated by the adaptive modal fusion weight calculation module are used to add the modal features in the cross-modal feature interaction module, so that the fusion weights of different modalities can be adaptively adjusted according to environmental changes, thereby being able to adapt to various complex and changing environments, thereby being able to solve the problem of inaccurate target detection caused by the weighted summation method using fixed weights in the prior art.

[0016] In order to make the above-mentioned objectives, features and advantages to be achieved by the embodiments of the present application more obvious and understandable, the following specifically cites preferred embodiments and describes them in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flow chart of a method for target detection based on multimodal fusion according to an adaptive weight provided in an embodiment of the present application is shown; Figure 2 A schematic diagram of the structure of a target detection model provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of the structure of any one of a first independent modal weight calculation module and a second independent modal weight calculation module provided in an embodiment of the present application is shown; Figure 4 A schematic diagram of the structure of a bias term calculation module provided in an embodiment of the present application is shown; Figure 5 A schematic diagram of the structure of any one of the first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule, or the third cross-modal feature interaction submodule provided in an embodiment of the present application is shown; Figure 6 A schematic diagram of the structure of a Transformer layer provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0020] In addition to the problem of inaccurate target detection caused by the use of fixed weights for weighted summation, the existing methods also have the following problems: Many existing target detection methods build complex models in dual-channel networks and then use basic fusion techniques such as concatenation or addition to enhance and integrate these multimodal information. Although this technology is simple to implement, its performance is limited in complex multimodal scenarios. For example, this method cannot fully utilize the diversity of information in visible light and infrared images, and often cannot effectively capture the more complex dependencies between the two modalities.

[0021] Based on this, the embodiment of the present application provides a target detection method for multimodal fusion based on adaptive weights, by obtaining infrared images and visible light images of the same scene taken by an infrared camera and a visible light camera, and then simultaneously inputting the infrared images and visible light images into a pre-trained target detection model for target detection, and obtaining target detection results. Among them, the target detection model includes a branch for extracting infrared image features from infrared images and another branch for extracting visible light image features from visible light images. For a given infrared image and visible light image, after feature extraction through a dual-stream network, the information of the dual modalities is integrated into three output streams of different sizes through a cross-modal feature interaction module at three different levels of the backbone network part of the target detection model. At the same time, the weights calculated by the adaptive modal fusion weight calculation module are used to add the modal features in the cross-modal feature interaction module, so that the fusion weights of different modalities can be adaptively adjusted according to environmental changes, and thus can adapt to various complex and changing environments, and thus can solve the problem of inaccurate target detection caused by the weighted summation method using fixed weights in the prior art.

[0022] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0023] See also Figure 1 , Figure 1 A flow chart of a target detection method for multimodal fusion based on adaptive weights provided in an embodiment of the present application is shown. It should be understood that the target detection method can be performed by an electronic device, and the specific device of the electronic device can be set according to actual needs, and the embodiment of the present application is not limited thereto. For example, the electronic device can be a computer, or a server, etc. Specifically, the target detection method includes: Step S110, acquiring an infrared image and a visible light image of the same scene captured by an infrared camera and a visible light camera.

[0024] It should be noted that, before executing step S120, the infrared image and the visible light image may be preprocessed. Moreover, the operations included in the preprocessing may be set according to actual needs, and the embodiment of the present application is not limited thereto.

[0025] For example, the preprocessing may include a binarization operation and a noise removal operation, wherein the binarization operation refers to performing a binarization operation on the infrared image and the visible light image; and the noise removal operation is to eliminate noise in the infrared image and the visible light image.

[0026] Step S120, simultaneously input the infrared image and the visible light image into a pre-trained target detection model to perform target detection, and obtain a target detection result, wherein the target detection result may include a detection box for selecting the target and a label for identifying the target type.

[0027] It should be understood that the specific network type and network structure of the target detection model can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0028] Alternatively, if Figure 2 As shown, when the target detection model is an improved YOLOv10 network, the target detection model includes a backbone network Backbone, a neck network Neck and a detection head Head.

[0029] Furthermore, the backbone network includes an infrared image feature extraction module for extracting infrared image features from infrared images, a visible light image feature extraction module for extracting visible light image features from visible light images, an adaptive modal fusion weight calculation module and a cross-modal feature interaction module.

[0030] Among them, the infrared image feature extraction module and the visible light image feature extraction module have the same structure, and the infrared image feature extraction module and the visible light image feature extraction module both include P1 to P5 layers, the P1 layer may be a convolution module Conv, and the P2 layer may include a convolution module and a C2f module connected in sequence, and the P3 layer may include a convolution module and a C2f module connected in sequence, and the P4 layer may include a downsampling SCDown module and a C2f module connected in sequence, and the P5 layer may include a downsampling SCDown module, a C2fCIB module, a spatial pyramid pooling SPPF module, and a PSA module connected in sequence; The adaptive modal fusion weight calculation module includes a first independent modal weight calculation module, a second independent modal weight calculation module, a bias item calculation module and a softmax layer, the first independent modal weight calculation module is used to determine a first preliminary weight value according to a feature map output by a P5 layer of an infrared image feature extraction module, and the second independent modal weight calculation module is used to determine a second preliminary weight value according to a feature map output by a P5 layer of a visible light image feature extraction module, and the bias item calculation module is used to determine a bias item according to a feature map output by a P3 layer of an infrared image feature extraction module and a feature map output by a P3 layer of a visible light image feature extraction module, and the softmax layer is used to determine a first modal fusion weight and a second modal fusion weight according to the first preliminary weight value, the second preliminary weight value and the bias item; The cross-modal feature interaction module is used to fuse corresponding feature maps according to the first modality fusion weight and the second modality fusion weight to obtain a fused feature map.

[0031] It should be understood that the structure of the first independent modal weight calculation module and the structure of the second independent modal weight calculation module can be set according to actual needs, and the embodiments of the present application are not limited thereto.

[0032] Alternatively, if Figure 3 As shown, when the structures of the first independent modal weight calculation module and the second independent modal weight calculation module are the same, the first independent modal weight calculation module and the second independent modal weight calculation module both include a first convolutional layer, a downsampling layer DownSample and a first multi-layer perception layer MLP connected in sequence.

[0033] For the first independent modality weight calculation module, the feature maps of the infrared modality are extracted by layers P1 to P5 in the infrared image feature extraction module, and then these feature maps are processed by a first convolution layer, and then downsampled by a downsampling layer, and then the first multi-layer perception layer MLP is used to calculate the first preliminary weight value of the single modality. w 1. Specifically: ; In the formula, w 1 represents the first preliminary weight value; FP 5- IR It represents the feature map of the infrared modality after passing through the P5 layer; * represents the convolution operation; k 1 represents the convolution kernel of the first convolutional layer in the first independent modality weight calculation module.

[0034] Similarly, the visible light mode also follows the same operation process as above to calculate its corresponding second preliminary weight value w 2. Specifically: ; In the formula, w 2 represents the second preliminary weight value; FP 5- RGB Represents the feature map of the visible light modality after passing through the P5 layer; k 2 represents the convolution kernel of the first convolution layer in the second independent modality weight calculation module.

[0035] It should also be understood that the structure of the bias item calculation module can also be set according to actual needs, and the embodiments of the present application are not limited to this.

[0036] Alternatively, if Figure 4 As shown, the bias item calculation module includes a second convolutional layer, a third convolutional layer, a multiplication operation layer, a subtraction operation layer, a fourth convolutional layer and a second multi-layer perception layer MLP, the second convolutional layer is respectively connected to the P3 layer, the multiplication operation layer and the subtraction operation layer of the infrared image feature extraction module, and the third convolutional layer is respectively connected to the P3 layer and the multiplication operation layer of the visible light image feature extraction module, and the multiplication operation layer is also connected to the subtraction operation layer, and the subtraction operation layer is also connected to the fourth convolutional layer, and the fourth convolutional layer is also connected to the second multi-layer perception layer.

[0037] In such Figure 4 Based on the structure shown in the figure, the dual-stream data input to the bias calculation module is the feature map of the two modalities after passing through the P3 layer, and the features of the two modalities are firstly used to highlight the salient object features in the multimodal image through the multiplication operation layer, and then the residual feature Residual Feature is obtained by subtracting the salient features from the features of the infrared modality, and then down-sampled by the fourth convolutional layer (which can be achieved by adjusting the step size), and finally the bias term is calculated using the second multi-layer perception layer MLP b , specifically: ; ; In the formula, b represents the bias term; R Residual Feature represents the residual feature obtained by subtracting the salient features from the features of the infrared modality; k 5 represents the convolution kernel of the fourth convolutional layer; FP 3- IR It represents the feature map of the infrared modality after passing through the P3 layer; k 3 represents the convolution kernel of the second convolutional layer; FP 3- RGB Represents the feature map of the visible light modality after passing through the P3 layer; × represents multiplication; k 4 represents the convolution kernel of the third convolutional layer.

[0038] Furthermore, the first preliminary weight value may be w 1. Second preliminary weight value w 2 and the bias term b pass soft Max regression is integrated together to obtain the final modal weight value WIR and WrG For multimodal feature fusion in the cross-modal feature interaction module, and the first modality fusion weight WIR The calculation expression is: ; In the formula, WIR represents the first modality fusion weight; soft max means soft max operation; w 1 represents the first preliminary weight value; w 2 represents the second preliminary weight value; b represents the bias term; And, the second modality fusion weight WrG The calculation expression is: ; In the formula, WrG Represents the second modality fusion weight.

[0039] It should also be understood that the structure of the cross-modal feature interaction module can also be set according to actual needs, and the embodiments of the present application are not limited to this.

[0040] Optionally, the cross-modal feature interaction module includes a first cross-modal feature interaction submodule A1, a second cross-modal feature interaction submodule A2 and a third cross-modal feature interaction submodule A3. Figure 1 As shown, the first cross-modal feature interaction submodule A1 is respectively connected to the softmax layer, the P3 layer of the infrared image feature extraction module, and the P3 layer of the visible light image feature extraction module, and the second cross-modal feature interaction submodule A2 is respectively connected to the softmax layer, the P4 layer of the infrared image feature extraction module, and the P4 layer of the visible light image feature extraction module, and the third cross-modal feature interaction submodule A3 is respectively connected to the softmax layer, the P5 layer of the infrared image feature extraction module, and the P5 layer of the visible light image feature extraction module.

[0041] Furthermore, if Figure 5As shown, when the structures of the first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule and the third cross-modal feature interaction submodule are the same, the first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule and the third cross-modal feature interaction submodule all include a fourth convolutional layer, a fifth convolutional layer, a first addition operation layer, a Transformer layer, a sixth convolutional layer, a seventh convolutional layer, a GSConv layer, an eighth convolutional layer, a second addition operation layer, a connection layer and a ninth convolutional layer, the first addition operation layer is connected to the fourth convolutional layer, the fifth convolutional layer and the Transformer layer, respectively, and the Transformer layer is also connected to the sixth convolutional layer and the seventh convolutional layer, respectively, and the sixth convolutional layer is also connected to the second addition operation layer, and the seventh convolutional layer is also connected to the GSConv layer and the eighth convolutional layer, and the second addition operation layer is also connected to the GSConv layer and the connection layer, and the connection layer is also connected to the eighth convolutional layer and the ninth convolutional layer.

[0042] And, if Figure 6 As shown, the Transformer layer includes a position encoding layer, a third addition operation layer, a first normalization layer, a multi-head attention layer, a fourth addition operation layer, a second normalization layer, a third multi-layer perception layer and a fifth addition operation layer, the third addition operation layer is respectively connected to the position encoding layer, the first normalization layer and the fourth addition operation layer, and the first normalization layer is also connected to the multi-head attention layer, and the multi-head attention layer is also connected to the fourth addition operation layer, and the fourth addition operation layer is respectively connected to the second normalization layer and the fifth addition operation layer, and the second normalization layer is also connected to the third multi-layer perception layer, and the third multi-layer perception layer is also connected to the fifth addition operation layer.

[0043] In such Figure 5 and Figure 6 Based on the structure shown in the figure, the information of the two streams is input and passed through a convolution layer respectively, and then multiplied by the weight coefficients corresponding to each modality (from the adaptive modality fusion weight calculation module). These weight coefficients are used to dynamically adjust the modality contribution, specifically: ; In the formula, F Represents the fused feature map; Fir The feature map representing the infrared modality is output after passing through the fourth convolutional layer; F The feature map representing the visible light modality is output after the fifth convolutional layer.

[0044] Subsequently, the target detection model processes the fused feature map through the Transformer layer. The self-attention mechanism of the Transformer layer can capture the long-distance dependencies in multimodal data, thereby more effectively integrating global context information and ensuring the complementarity of information between different modalities. Next, the fused features are extracted using the lightweight GSConv. GSConv uses efficient convolution operations to reduce the amount of calculation while maintaining good feature extraction capabilities. Finally, the features of different levels and paths are deeply integrated using skip connections, addition, and Concat.

[0045] It should be noted here that, based on the backbone network Backbone of the target detection model, the neck network Neck can output the target detection result based on the first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule and the third cross-modal feature interaction submodule.

[0046] Therefore, with the help of the above technical solution, the present application obtains infrared images and visible light images of the same scene taken by an infrared camera and a visible light camera, and then simultaneously inputs the infrared images and visible light images into a pre-trained target detection model for target detection to obtain target detection results. Among them, the target detection model includes a branch for extracting infrared image features from infrared images and another branch for extracting visible light image features from visible light images. For a given infrared image and visible light image, after feature extraction by a dual-stream network, the information of the two modalities is integrated into three output streams of different sizes through a cross-modal feature interaction module at three different levels of the backbone network part of the target detection model. At the same time, the weights calculated by the adaptive modal fusion weight calculation module are used to add the modal features in the cross-modal feature interaction module, so that the fusion weights of different modalities can be adaptively adjusted according to environmental changes, and thus can adapt to various complex and changing environments, and thus can solve the problem of inaccurate target detection caused by the weighted summation method using fixed weights in the prior art.

[0047] It should be understood that the above-mentioned target detection method for multimodal fusion based on adaptive weights is only exemplary, and those skilled in the art can make various modifications based on the above-mentioned method, and the modified scheme also belongs to the protection scope of this application.

[0048] The present application provides a storage medium having a computer program stored thereon. The computer program is executed by a processor to execute the method described in the embodiment.

[0049] The present application provides an electronic device, including a processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the method embodiment.

[0050] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0051] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0052] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0053] It should be noted that the word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising a number of different components and by means of a suitably programmed computer. Among the several means listed, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the name of the component.

[0054] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0055] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the technical solution should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0056] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the technical solution of the present invention and its equivalent technology, the present invention should also include these modifications and variations.

Claims

1. A target detection method for multimodal fusion based on adaptive weights, characterized in that: include: Obtain infrared images and visible light images of the same scene taken by an infrared camera and a visible light camera; At the same time, the infrared image and the visible light image are input into a pre-trained target detection model to perform target detection and obtain a target detection result; Among them, the target detection model includes a backbone network and a neck network; the backbone network includes an infrared image feature extraction module, a visible light image feature extraction module, an adaptive modal fusion weight calculation module and a cross-modal feature interaction module; the infrared image feature extraction module and the visible light image feature extraction module both include P1 layer to P5 layer; the adaptive modal fusion weight calculation module includes a first independent modality weight calculation module, a second independent modality weight calculation module, a bias item calculation module and a softmax layer, the first independent modality weight calculation module is used to determine the first preliminary weight value according to the feature map output by the P5 layer of the infrared image feature extraction module, and the second independent modality weight calculation module is used to determine the first preliminary weight value according to the visible light image feature extraction module. The second preliminary weight value is determined according to the feature map output by the P5 layer of the visible light image feature extraction module, and the bias item calculation module is used to determine the bias item according to the feature map output by the P3 layer of the infrared image feature extraction module and the feature map output by the P3 layer of the visible light image feature extraction module, and the softmax layer is used to determine the first modality fusion weight and the second modality fusion weight according to the first preliminary weight value, the second preliminary weight value and the bias item; the cross-modal feature interaction module is used to fuse the corresponding feature maps according to the first modality fusion weight and the second modality fusion weight to obtain a fused feature map; the neck network is used to output the target detection result according to the fused feature map.

2. The target detection method according to claim 1, characterized in that: The first independent modal weight calculation module and the second independent modal weight calculation module have the same structure; The first independent modality weight calculation module and the second independent modality weight calculation module both include a first convolutional layer, a downsampling layer and a first multi-layer perception layer connected in sequence.

3. The target detection method according to claim 1 or 2, characterized in that: The bias term calculation module includes a second convolution layer, a third convolution layer, a multiplication operation layer, a subtraction operation layer, a fourth convolution layer and a second multi-layer perception layer, the second convolution layer is respectively connected to the P3 layer of the infrared image feature extraction module, the multiplication operation layer and the subtraction operation layer, and the third convolution layer is respectively connected to the P3 layer and the multiplication operation layer of the visible light image feature extraction module, and the multiplication operation layer is also connected to the subtraction operation layer, and the subtraction operation layer is also connected to the fourth convolution layer, and the fourth convolution layer is also connected to the second multi-layer perception layer.

4. The target detection method according to claim 1, characterized in that: The cross-modal feature interaction module includes a first cross-modal feature interaction sub-module, a second cross-modal feature interaction sub-module and a third cross-modal feature interaction sub-module, the first cross-modal feature interaction sub-module is respectively connected to the softmax layer, the P3 layer of the infrared image feature extraction module and the P3 layer of the visible light image feature extraction module, and the second cross-modal feature interaction sub-module is respectively connected to the softmax layer, the P4 layer of the infrared image feature extraction module and the P4 layer of the visible light image feature extraction module, and the third cross-modal feature interaction sub-module is respectively connected to the softmax layer, the P5 layer of the infrared image feature extraction module and the P5 layer of the visible light image feature extraction module.

5. The target detection method according to claim 4, characterized in that: The first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule and the third cross-modal feature interaction submodule have the same structure; The first cross-modal feature interaction submodule, the second cross-modal feature interaction submodule and the third cross-modal feature interaction submodule all include a fourth convolutional layer, a fifth convolutional layer, a first addition operation layer, a Transformer layer, a sixth convolutional layer, a seventh convolutional layer, a GSConv layer, an eighth convolutional layer, a second addition operation layer, a connection layer and a ninth convolutional layer, the first addition operation layer is connected to the fourth convolutional layer, the fifth convolutional layer and the Transformer layer, respectively, and the Transformer layer is also connected to the sixth convolutional layer and the seventh convolutional layer, respectively, and the sixth convolutional layer is also connected to the second addition operation layer, and the seventh convolutional layer is also connected to the GSConv layer and the eighth convolutional layer, and the second addition operation layer is also connected to the GSConv layer and the connection layer, and the connection layer is also connected to the eighth convolutional layer and the ninth convolutional layer.

6. The target detection method according to claim 5, characterized in that: The Transformer layer includes a position encoding layer, a third addition operation layer, a first normalization layer, a multi-head attention layer, a fourth addition operation layer, a second normalization layer, a third multi-layer perception layer and a fifth addition operation layer. The third addition operation layer is respectively connected to the position encoding layer, the first normalization layer and the fourth addition operation layer, and the first normalization layer is connected to the multi-head attention layer, and the multi-head attention layer is also connected to the fourth addition operation layer, and the fourth addition operation layer is respectively connected to the second normalization layer and the fifth addition operation layer, and the second normalization layer is also connected to the third multi-layer perception layer, and the third multi-layer perception layer is also connected to the fifth addition operation layer.

7. The target detection method according to claim 1, characterized in that: The calculation expression of the first modality fusion weight is: ； In the formula, WIR represents the first modality fusion weight; soft max means soft max operation; w 1 represents the first preliminary weight value; w 2 represents the second preliminary weight value; b represents the bias term; And, the calculation expression of the second modality fusion weight is: ； In the formula, WrG Represents the second modality fusion weight.

8. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target detection method for performing multimodal fusion according to adaptive weights as described in any one of claims 1 to 7 are executed.

9. An electronic device comprising a processor, a memory and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the target detection method for performing multimodal fusion according to adaptive weights as described in any one of claims 1-7.

Images