A target detection method based on adaptive fusion of visible light and infrared features

Abstract

The present application provides a target detection method based on adaptive fusion of visible light and infrared features, which relates to the technical field of target detection. The method includes: performing channel attention processing and spatial attention processing on visible light feature maps and infrared feature maps, and obtaining a first fused feature map based on the spatial attention processing result; performing coordinate attention processing and calculating visible light feature weights and infrared feature weights based on the spatial attention processing result; obtaining a second fused feature map based on the coordinate attention processing result and the weight calculation result; obtaining a third fused feature map based on the second fused feature map; performing multi-scale feature fusion based on the first fused feature map, the second fused feature map and the third fused feature map; and performing target detection based on the multi-scale feature fusion result. Compared with existing single-modal and multi-modal target detection algorithms, the method provided by the present application can obtain higher detection accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a target detection method based on adaptive fusion of visible light and infrared features. Background Art

[0002] With the development of artificial intelligence and the improvement of hardware level, target detection technology based on deep learning has become the focus of remote sensing. However, most of the current target detection methods are mainly based on visible light images, which use the ambient light reflected by the target to form images. Although they can fully display the details and texture features of the target, it is difficult to fully obtain target information by visible light sensors alone when the target is blurred, blocked, or the light is too strong or too dark, thus affecting the target detection effect. In contrast, infrared sensors have the advantages of strong resistance to environmental interference, insensitivity to light, and the ability to reflect the temperature of the target, which can make up for the shortcomings of visible light imaging. Therefore, integrating infrared features into target detection based on visible light and studying target detection algorithms with synergistic effects of multi-sensor information can not only improve the accuracy and robustness of target detection, but also be better suited for actual scenes. However, infrared images themselves also have disadvantages such as low resolution and unclear target details. Under conditions of abundant ambient light, the detection accuracy is not as good as that of visible light images.

[0003] Therefore, how to adaptively fuse visible light and infrared modalities based on their complementary characteristics is a hot topic and difficulty in current research on multimodal target detection tasks. Summary of the invention

[0004] The embodiments of the present application provide a target detection method based on adaptive fusion of visible light and infrared features to solve the technical problems existing in the prior art.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, a target detection method based on adaptive fusion of visible light and infrared features is provided, comprising:

[0007] Obtaining a visible light feature map based on the visible light image and obtaining an infrared feature map based on the infrared image;

[0008] performing channel attention processing and spatial attention processing on the visible light feature map and the infrared feature map in sequence, and obtaining a first fused feature map based on the spatial attention processing result;

[0009] Based on the results of spatial attention processing, coordinate attention processing is performed and visible light feature weights and infrared feature weights are calculated;

[0010] A second fused feature map is obtained based on the coordinate attention processing result and the weight calculation result;

[0011] Obtain a third fused feature map based on the second fused feature map;

[0012] Perform multi-scale feature fusion based on the first fused feature map, the second fused feature map, and the third fused feature map;

[0013] Target detection is performed based on multi-scale feature fusion results.

[0014] In some embodiments of the present application, based on the aforementioned solution, the channel attention processing and the spatial attention processing are performed on the visible light feature map and the infrared feature map in sequence, and a first fused feature map is obtained based on the spatial attention processing result, including:

[0015] Performing channel attention processing on the visible light feature map and the infrared feature map respectively to obtain a first-reconstructed visible light feature map and a first-reconstructed infrared feature map;

[0016] Performing spatial attention processing on the first reconstructed visible light feature map and the first reconstructed infrared feature map respectively to obtain a second reconstructed visible light feature map and a second reconstructed infrared feature map;

[0017] The second reconstructed visible light feature map and the second reconstructed infrared feature map are spliced ​​and fused to obtain the first fused feature map.

[0018] In some embodiments of the present application, based on the aforementioned solution, the channel attention processing is performed on the visible light feature map and the infrared feature map respectively to obtain a reconstructed visible light feature map and a reconstructed infrared feature map, including:

[0019] The spatial features of the visible light feature map and the infrared feature map are aggregated respectively by average pooling and maximum pooling to generate two different spatial feature descriptors, and a visible light feature channel attention parameter and an infrared feature channel attention parameter are generated by a multi-layer perceptron;

[0020] Multiplying the visible light feature channel attention parameter by the visible light feature map channel by channel to obtain the primary reconstructed visible light feature map;

[0021] The infrared feature channel attention parameter is multiplied by the infrared feature map channel by channel to obtain the primary reconstructed infrared feature map.

[0022] In some embodiments of the present application, based on the aforementioned scheme, the spatial attention processing is performed on the first reconstructed visible light feature map and the first reconstructed infrared feature map to obtain a second reconstructed visible light feature map and a second reconstructed infrared feature map, including:

[0023] Aggregating the channel information of the first reconstructed visible light feature map and the first reconstructed infrared feature map by average pooling and maximum pooling respectively, to obtain an aggregated first reconstructed visible light feature map and a first reconstructed infrared feature map;

[0024] The aggregated first reconstructed visible light feature map and the first reconstructed infrared feature map are spliced ​​and dimensionality reduced by convolution to obtain a first reconstructed visible light feature map and a first reconstructed infrared feature map after dimensionality reduction;

[0025] Based on the first reconstructed visible light feature map and the first reconstructed infrared feature map after dimensionality reduction, the spatial attention feature is activated by the Sigmoid function;

[0026] The spatial attention feature is subjected to dot multiplication processing with the first-reconstructed visible light feature map and the first-reconstructed infrared feature map respectively to obtain the second-reconstructed visible light feature map and the second-reconstructed infrared feature map.

[0027] In some embodiments of the present application, based on the aforementioned scheme, coordinate attention processing is performed based on the spatial attention processing result, including:

[0028] Coordinate attention processing is performed on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively to obtain a tertiary reconstructed visible light feature map and a tertiary reconstructed infrared feature map.

[0029] In some embodiments of the present application, based on the aforementioned scheme, the coordinate attention processing is performed on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively to obtain a third reconstructed visible light feature map and a third reconstructed infrared feature map, including:

[0030] Performing coordinate attention processing on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively;

[0031] The tertiary reconstructed visible light feature map and the tertiary reconstructed infrared feature map are obtained based on the coordinate attention processing result and the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map.

[0032] In some embodiments of the present application, based on the above scheme, the visible light feature weight and the infrared feature weight are calculated based on the spatial attention processing result, including:

[0033] The secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map are connected to obtain weight features, and the weight features are passed through a multi-layer perceptron to obtain a fusion weight of the visible light feature and a fusion weight of the infrared feature;

[0034] Extract ambient light intensity weight;

[0035] The visible light feature weight is calculated based on the ambient light intensity weight and the fusion weight of the visible light feature;

[0036] The infrared feature weight is calculated based on the ambient light intensity weight and the fusion weight of the infrared feature.

[0037] In some embodiments of the present application, based on the aforementioned solution, the second fused feature map is obtained based on the coordinate attention processing result and the weight calculation result, including:

[0038] Multiplying the visible light feature weight by the three-times reconstructed visible light feature map, and multiplying the infrared feature weight by the three-times reconstructed infrared feature map;

[0039] The two multiplication results are concatenated by channel to obtain the second fused feature map.

[0040] In some embodiments of the present application, based on the aforementioned solution, the multi-scale feature fusion based on the first fused feature map, the second fused feature map and the third fused feature map includes:

[0041] The first fusion feature map, the second fusion feature map and the third fusion feature map are upsampled and fused and fused in a bottom-up path to obtain a first detection feature map, a second detection feature map and a third detection feature map.

[0042] In some embodiments of the present application, based on the above-mentioned solution, the target detection based on the multi-scale feature fusion result includes:

[0043] Target detection is performed on the first detection feature map, the second detection feature map and the third detection feature map respectively to obtain detection results, and non-maximum suppression processing is performed on the detection results.

[0044] According to a second aspect of an embodiment of the present application, there is provided a target detection device based on adaptive fusion of visible light and infrared features, comprising:

[0045] An acquisition unit, used to obtain a visible light feature map based on the visible light image and to obtain an infrared feature map based on the infrared image;

[0046] A first processing unit is used to perform channel attention processing and spatial attention processing on the visible light feature map and the infrared feature map in sequence, and obtain a first fused feature map based on the spatial attention processing result;

[0047] A second processing unit is used to perform coordinate attention processing and calculate visible light feature weights and infrared feature weights based on the spatial attention processing result;

[0048] A third processing unit, configured to obtain a second fused feature map based on the coordinate attention processing result and the weight calculation result;

[0049] A fourth processing unit, configured to obtain a third fused feature map based on the second fused feature map;

[0050] A fusion unit, configured to perform multi-scale feature fusion based on the first fused feature map, the second fused feature map, and the third fused feature map;

[0051] The detection unit is used to perform target detection based on the multi-scale feature fusion result.

[0052] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method according to the first aspect.

[0053] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including a memory and a processor;

[0054] The memory is used to store computer instructions;

[0055] The processor is used to call the computer instructions stored in the memory so that the electronic device executes the method as described in the first aspect.

[0056] The technical solution of the present application extracts the complementary features of visible light and infrared images respectively, conducts joint channel and spatial attention analysis, realizes the exchange and reorganization of cross-modal information weights, and improves the perception ability of complementary features between different modalities; at the same time, it introduces light perception to guide adaptive fusion, which solves the problem that conventional fusion methods are difficult to achieve dynamic processing based on multimodal data features.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0059] Figure 1 A schematic flow chart of a target detection method based on adaptive fusion of visible light and infrared features according to an embodiment of the present application is shown;

[0060] Figure 2 A block diagram of a target detection device based on adaptive fusion of visible light and infrared features according to an embodiment of the present application is shown;

[0061] Figure 3 A block diagram of an electronic device according to an embodiment of the present application is shown;

[0062] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0064] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0065] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0066] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0067] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0068] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] Some embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0070] In order to solve the technical problems in the prior art, the embodiment of the present application provides a target detection method based on the adaptive fusion of visible light and infrared features. In the embodiment of the present application, the method is implemented by a dual-branch multimodal target detection model. Specifically, the structure of the dual-branch multimodal target detection model includes a backbone network, a neck network and a detection head. Among them, the backbone network is mainly used to extract the features of visible light images and infrared images, and uses a fusion module to fuse the extracted features at multiple scales; the neck network uses multi-scale feature fusion to fuse the feature maps of different stages of the backbone network to enhance the feature expression capability; the detection head part is responsible for further processing the features output by the neck network part, and converting the feature map into specific information required for target detection, including category, location and confidence, to generate the final detection result.

[0071] See also Figure 1 , shows a flow chart of a target detection method based on adaptive fusion of visible light and infrared features according to an embodiment of the present application.

[0072] like Figure 1 As shown, a target detection method based on adaptive fusion of visible light and infrared features is demonstrated, which specifically includes steps S100 to S700.

[0073] refer to Figure 1 , step S100, obtaining a visible light feature map based on the visible light image and obtaining an infrared feature map based on the infrared image.

[0074] Exemplarily, the process of obtaining the visible light feature map is as follows:

[0075] The visible light image is processed by convolution + batch normalization + activation function, and then feature enhancement and fusion convolution are performed to obtain the visible light feature map.

[0076] Similarly, the process of obtaining the infrared feature map is as follows:

[0077] The infrared image is also processed by convolution + batch normalization + activation function, and then feature enhancement and fusion convolution are performed to obtain the infrared feature map.

[0078] Continue to refer Figure 1 , step S200, performing channel attention processing and spatial attention processing on the visible light feature map and the infrared feature map in turn, and obtaining a first fused feature map based on the spatial attention processing result.

[0079] In some feasible embodiments, based on the above solution, step S200 includes:

[0080] Step S210, performing channel attention processing on the visible light feature map and the infrared feature map respectively to obtain a primary reconstructed visible light feature map and a primary reconstructed infrared feature map;

[0081] Step S220, performing spatial attention processing on the first reconstructed visible light feature map and the first reconstructed infrared feature map respectively to obtain a second reconstructed visible light feature map and a second reconstructed infrared feature map;

[0082] Step S230: splicing and fusing the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map to obtain the first fused feature map.

[0083] In some feasible embodiments, based on the above solution, step S210 includes:

[0084] The spatial features of the visible light feature map and the infrared feature map are aggregated respectively by average pooling and maximum pooling to generate two different spatial feature descriptors, and a visible light feature channel attention parameter and an infrared feature channel attention parameter are generated by a multi-layer perceptron;

[0085] Multiplying the visible light feature channel attention parameter by the visible light feature map channel by channel to obtain the primary reconstructed visible light feature map;

[0086] The infrared feature channel attention parameter is multiplied by the infrared feature map channel by channel to obtain the primary reconstructed infrared feature map.

[0087] In some feasible embodiments, based on the above solution, step S220 includes:

[0088] Aggregating the channel information of the first reconstructed visible light feature map and the first reconstructed infrared feature map by average pooling and maximum pooling respectively, to obtain an aggregated first reconstructed visible light feature map and a first reconstructed infrared feature map;

[0089] The aggregated first reconstructed visible light feature map and the first reconstructed infrared feature map are spliced ​​and dimensionality reduced by convolution to obtain a first reconstructed visible light feature map and a first reconstructed infrared feature map after dimensionality reduction;

[0090] Based on the first reconstructed visible light feature map and the first reconstructed infrared feature map after dimensionality reduction, the spatial attention feature is activated by the Sigmoid function;

[0091] The spatial attention feature is subjected to dot multiplication processing with the first-reconstructed visible light feature map and the first-reconstructed infrared feature map respectively to obtain the second-reconstructed visible light feature map and the second-reconstructed infrared feature map.

[0092] Continue to refer Figure 1 , step S300, coordinate attention processing is performed based on the spatial attention processing result, and visible light feature weights and infrared feature weights are calculated.

[0093] In some feasible embodiments, based on the above scheme, coordinate attention processing is performed based on the spatial attention processing result, including:

[0094] Coordinate attention processing is performed on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively to obtain a tertiary reconstructed visible light feature map and a tertiary reconstructed infrared feature map.

[0095] In some feasible embodiments, based on the above scheme, the coordinate attention processing is performed on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively to obtain the third reconstructed visible light feature map and the third reconstructed infrared feature map, including:

[0096] Performing coordinate attention processing on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively;

[0097] The tertiary reconstructed visible light feature map and the tertiary reconstructed infrared feature map are obtained based on the coordinate attention processing result and the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map.

[0098] Exemplarily, after receiving the second reconstructed visible light feature map and the second reconstructed infrared feature map, coordinate attention is used to highlight the feature information of each of the two feature maps, and the processed results are added to the input features layer by layer. After convolution, the residual structure is used to add the second reconstructed visible light feature map and the second reconstructed infrared feature map to obtain the third reconstructed visible light feature map and the third reconstructed infrared feature map.

[0099] In some feasible embodiments, based on the above scheme, the visible light feature weight and the infrared feature weight are calculated based on the spatial attention processing result, including:

[0100] The secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map are connected to obtain weight features, and the weight features are passed through a multi-layer perceptron to obtain a fusion weight of the visible light feature and a fusion weight of the infrared feature;

[0101] Extract ambient light intensity weight;

[0102] The visible light feature weight is calculated based on the ambient light intensity weight and the fusion weight of the visible light feature;

[0103] The infrared feature weight is calculated based on the ambient light intensity weight and the fusion weight of the infrared feature.

[0104] Exemplarily, the calculation process of the visible light feature weight is as follows:

[0105] The ambient light intensity weight ωillu is regarded as the light weight corresponding to the visible light feature;

[0106] The visible light feature weight is obtained by adding the illumination weight ωillu of the visible light feature and the fusion weight ω1 of the visible light feature.

[0107] The calculation process of infrared feature weight is as follows:

[0108] Consider 1-ωillu as the illumination weight corresponding to the infrared feature;

[0109] The infrared feature weight is obtained by adding the illumination weight 1-ωillu of the infrared feature and the fusion weight ω2 of the infrared feature.

[0110] Continue to refer Figure 1 , step S400, obtaining a second fused feature map based on the coordinate attention processing result and the weight calculation result.

[0111] In some feasible embodiments, based on the above scheme, the second fused feature map is obtained based on the coordinate attention processing result and the weight calculation result, including:

[0112] Multiplying the visible light feature weight by the three-times reconstructed visible light feature map, and multiplying the infrared feature weight by the three-times reconstructed infrared feature map;

[0113] The two multiplication results are concatenated by channel to obtain the second fused feature map.

[0114] Continue to refer Figure 1 , step S500, obtaining a third fused feature map based on the second fused feature map.

[0115] Exemplarily, the second fused feature map is subjected to two convolutions and one spatial pyramid pooling to obtain a third fused feature map.

[0116] Continue to refer Figure 1 , step S600, performing multi-scale feature fusion based on the first fused feature map, the second fused feature map and the third fused feature map.

[0117] In some feasible embodiments, based on the above solution, the multi-scale feature fusion based on the first fused feature map, the second fused feature map and the third fused feature map includes:

[0118] The first fusion feature map, the second fusion feature map and the third fusion feature map are upsampled and fused and fused in a bottom-up path to obtain a first detection feature map, a second detection feature map and a third detection feature map.

[0119] Exemplarily, the specific process of multi-scale feature fusion is as follows:

[0120] The third fused feature map is upsampled by 2 times to obtain a feature map of size 40x40. The upsampled feature map is fused with the second fused feature map to obtain feature map F1 (40x40). Then feature map F1 undergoes a convolution and upsampling by 2 times operation to obtain a feature map of size 80x80, which is then fused with the first fused feature map to obtain the first detection feature map. (80x80).

[0121] The first detection feature map After two convolution operations, the feature map size is 40x40. The convolution feature map is fused with the feature map F1 (40x40) to obtain the feature map F2 (40x40). The feature map F2 is processed by the feature enhancement and fusion convolution module to obtain the first detection feature map. (40x40). The first detection feature map After two convolution operations, the feature map size is 20x20. The convolved feature map is fused with the third fused feature map (20x20) to obtain the feature map F3 (20x20). The feature map F3 is processed by the feature enhancement and fusion convolution module to obtain the first detection feature map (20x20).

[0122] Continue to refer Figure 1 , step S700, performing target detection based on the multi-scale feature fusion result.

[0123] In some feasible embodiments, based on the above solution, the target detection based on the multi-scale feature fusion result includes:

[0124] Target detection is performed on the first detection feature map, the second detection feature map and the third detection feature map respectively to obtain detection results, and non-maximum suppression processing is performed on the detection results.

[0125] Exemplarily, the specific process of target detection is as follows:

[0126] The first detection feature map Take target detection as an example:

[0127] After convolution and prediction processing, the first detection feature map is converted into a target detection result. The target detection result mainly consists of three parts:

[0128] 1. Bounding box regression: predict the location and size of the target. Usually output four values, corresponding to the center coordinates (x, y) and width and height (w, h) of the bounding box.

[0129] 2. Confidence score: predict whether each bounding box contains an object and the confidence of the object.

[0130] 3. Category probability: predict the probability that the target belongs to each category.

[0131] The final prediction result will be processed by non-maximum suppression to remove duplicate detection boxes. Non-maximum suppression retains the bounding box with the highest confidence and removes other bounding boxes with high overlap to ensure that each object is detected only once.

[0132] Similarly, the second detection feature map And the third detection feature map The processing steps are consistent with the above process. The first detection feature map The size of 80x80 is specially used to detect small objects. The second detection feature map The size of 40x40 is specifically used to detect medium-sized objects. The third detection feature map The size of 20x20 is specially used to detect large objects. Multi-scale object detection is achieved through feature maps of different sizes.

[0133] The following describes an embodiment of the device of the present application, which can be used to execute a target detection method based on adaptive fusion of visible light and infrared features in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the above method of the present application.

[0134] Reference Figure 2 As shown, according to an embodiment of the present application, a target detection device 200 based on adaptive fusion of visible light and infrared features includes:

[0135] An acquisition unit 201 is used to obtain a visible light feature map based on the visible light image and to obtain an infrared feature map based on the infrared image;

[0136] A first processing unit 202 is used to perform channel attention processing and spatial attention processing on the visible light feature map and the infrared feature map in sequence, and obtain a first fused feature map based on the spatial attention processing result;

[0137] The second processing unit 203 is used to perform coordinate attention processing and calculate visible light feature weights and infrared feature weights based on the spatial attention processing result;

[0138] The third processing unit 204 is used to obtain a second fused feature map based on the coordinate attention processing result and the weight calculation result;

[0139] A fourth processing unit 205, configured to obtain a third fused feature map based on the second fused feature map;

[0140] A fusion unit 206, configured to perform multi-scale feature fusion based on the first fused feature map, the second fused feature map, and the third fused feature map;

[0141] The detection unit 207 is used to perform target detection based on the multi-scale feature fusion result.

[0142] like Figure 3 As shown, an embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, the steps of the above-mentioned target detection method based on adaptive fusion of visible light and infrared features are implemented.

[0143] Since the electronic device introduced in this embodiment is a device used to implement a target detection device based on adaptive fusion of visible light and infrared features in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation mode of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment adopted by technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0144] During the specific implementation process, when the computer program 311 is executed by a processor, any implementation method in the embodiments corresponding to the first aspect can be implemented.

[0145] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0146] It should be noted that Figure 4 The computer system 400 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0147] like Figure 4 As shown, the computer system 400 includes a central processing unit 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory 402 or the program loaded from the storage part 408 into the random access memory 403, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the random access memory 403. The central processing unit 401, the read-only memory 402 and the random access memory 403 are connected to each other via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0148] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0149] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the system of the present application are executed.

[0150] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0151] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0153] As another aspect, the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a target detection method based on adaptive fusion of visible light and infrared features described in the above embodiment.

[0154] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements a target detection method based on adaptive fusion of visible light and infrared features described in the above embodiment.

[0155] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0156] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0157] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. It should be understood that the present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A target detection method based on adaptive fusion of visible light and infrared features, characterized in that: include: Obtaining a visible light feature map based on the visible light image and obtaining an infrared feature map based on the infrared image; performing channel attention processing and spatial attention processing on the visible light feature map and the infrared feature map in sequence, and obtaining a first fused feature map based on the spatial attention processing result; Based on the results of spatial attention processing, coordinate attention processing is performed and visible light feature weights and infrared feature weights are calculated; A second fused feature map is obtained based on the coordinate attention processing result and the weight calculation result; Obtain a third fused feature map based on the second fused feature map; Perform multi-scale feature fusion based on the first fused feature map, the second fused feature map, and the third fused feature map; Target detection based on multi-scale feature fusion results; The sequentially performing channel attention processing and spatial attention processing on the visible light feature map and the infrared feature map, and obtaining a first fused feature map based on the spatial attention processing result, includes: Performing channel attention processing on the visible light feature map and the infrared feature map respectively to obtain a first-reconstructed visible light feature map and a first-reconstructed infrared feature map; Performing spatial attention processing on the first reconstructed visible light feature map and the first reconstructed infrared feature map respectively to obtain a second reconstructed visible light feature map and a second reconstructed infrared feature map; The second reconstructed visible light feature map and the second reconstructed infrared feature map are spliced ​​and fused to obtain the first fused feature map; Coordinate attention processing is performed based on the results of spatial attention processing, including: Performing coordinate attention processing on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively to obtain a tertiary reconstructed visible light feature map and a tertiary reconstructed infrared feature map; The visible light feature weight and infrared feature weight are calculated based on the spatial attention processing results, including: The secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map are connected to obtain weight features, and the weight features are passed through a multi-layer perceptron to obtain a fusion weight of the visible light feature and a fusion weight of the infrared feature; Extract ambient light intensity weight; The visible light feature weight is calculated based on the ambient light intensity weight and the fusion weight of the visible light feature; The infrared feature weight is calculated based on the ambient light intensity weight and the fusion weight of the infrared feature; The second fused feature map is obtained based on the coordinate attention processing result and the weight calculation result, including: Multiplying the visible light feature weight by the three-times reconstructed visible light feature map, and multiplying the infrared feature weight by the three-times reconstructed infrared feature map; The two multiplication results are concatenated by channel to obtain the second fused feature map.

2. The method according to claim 1, characterized in that The performing channel attention processing on the visible light feature map and the infrared feature map respectively to obtain a reconstructed visible light feature map and a reconstructed infrared feature map includes: The spatial features of the visible light feature map and the infrared feature map are aggregated respectively by average pooling and maximum pooling to generate two different spatial feature descriptors, and a visible light feature channel attention parameter and an infrared feature channel attention parameter are generated by a multi-layer perceptron; Multiplying the visible light feature channel attention parameter by the visible light feature map channel by channel to obtain the primary reconstructed visible light feature map; The infrared feature channel attention parameter is multiplied by the infrared feature map channel by channel to obtain the primary reconstructed infrared feature map.

3. The method according to claim 1, characterized in that The performing spatial attention processing on the first reconstructed visible light feature map and the first reconstructed infrared feature map respectively to obtain a second reconstructed visible light feature map and a second reconstructed infrared feature map comprises: Aggregating the channel information of the first reconstructed visible light feature map and the first reconstructed infrared feature map by average pooling and maximum pooling respectively, to obtain an aggregated first reconstructed visible light feature map and a first reconstructed infrared feature map; The aggregated first reconstructed visible light feature map and the first reconstructed infrared feature map are spliced ​​and dimensionality reduced by convolution to obtain a first reconstructed visible light feature map and a first reconstructed infrared feature map after dimensionality reduction; Based on the first reconstructed visible light feature map and the first reconstructed infrared feature map after dimensionality reduction, the spatial attention feature is activated by the Sigmoid function; The spatial attention feature is subjected to dot multiplication processing with the first-reconstructed visible light feature map and the first-reconstructed infrared feature map respectively to obtain the second-reconstructed visible light feature map and the second-reconstructed infrared feature map.

4. The method according to claim 1, characterized in that: The coordinate attention processing is performed on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively to obtain a tertiary reconstructed visible light feature map and a tertiary reconstructed infrared feature map, including: Performing coordinate attention processing on the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map respectively; The tertiary reconstructed visible light feature map and the tertiary reconstructed infrared feature map are obtained based on the coordinate attention processing result and the secondary reconstructed visible light feature map and the secondary reconstructed infrared feature map.

5. The method according to claim 1, characterized in that: The performing multi-scale feature fusion based on the first fused feature map, the second fused feature map, and the third fused feature map includes: The first fusion feature map, the second fusion feature map and the third fusion feature map are upsampled and fused and fused in a bottom-up path to obtain a first detection feature map, a second detection feature map and a third detection feature map.

6. The method according to claim 5, characterized in that The target detection based on the multi-scale feature fusion result includes: Target detection is performed on the first detection feature map, the second detection feature map and the third detection feature map respectively to obtain detection results, and non-maximum suppression processing is performed on the detection results.

Images