Dual-light image detection method and target detector

Abstract

The invention relates to a dual-light image detection method and a target detector. The method comprises the following steps: respectively inputting a received visible light image and an infrared image into a first backbone network and a second backbone network of the target detector; respectively extracting features of the visible light image and the infrared image through a first backbone network and a second backbone network to obtain feature information of a first mode and a second mode; fusing the feature information of the first mode and the second mode through a fusion network module of the target detector to obtain fusion features; and obtaining classification and positioning information of the input image according to the fusion features. According to the scheme of the invention, the network model of the target detector is improved, the feature level fusion module is introduced, the first mode and the second mode can be marked at the same time, and the problem of misalignment of double lights is solved by using the feature level fusion implicit alignment feature; in addition, decision-level fusion is performed after feature-level fusion, and the accuracy of a detection result can be improved by using two different fusion modes.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a dual-light image detection method and a target detector. Background Art

[0002] Multimodal drone perception technology based on visible light (RGB, Red, Green and Blue) images and infrared (IR, Infrared Radiation) images has a wide range of applications, such as in agriculture, environmental monitoring, urban planning and other fields. Ideally, the use of multimodal data is conducive to improving perception accuracy. However, the complexity of the imaging background and the inconsistency of multimodal data will increase the difficulty of perception, resulting in the multimodal drone perception technology not having obvious advantages over single-modal perception technology. For complex aerial photography scenes, the detection effect is poor mainly because the existing methods are difficult to accurately capture the details and contextual information of the target, resulting in a decrease in detection accuracy. In addition, during the movement of the drone, there will be a significant time difference in the acquisition of the perception image, which makes the dual-light data usually unable to remain aligned.

[0003] Therefore, object detection from an aerial perspective faces two main challenges: the lack of high-precision detection models and the mismatch of multimodal images. Summary of the invention

[0004] The present application has conducted relevant research on the above-mentioned problems of the prior art and proposed an effective solution. At present, the dual-light image detection method is mainly divided into three types of detection fusion methods, namely image level fusion, feature level fusion and decision level fusion. Among them, feature fusion can maintain implicit alignment, and at the same time learn the position and category information of the first modality and the second modality. Therefore, based on feature fusion, the present application provides an end-to-end efficient detection method. In view of the problem of unpaired visible light images and infrared images, a fusion module as well as feature level and decision level fusion detection strategies are designed, so that it can accurately identify the target position in the image and the target classification information.

[0005] According to a first aspect of the present application, a dual-light image detection method is provided, characterized in that it includes:

[0006] receiving an input image, wherein the input image includes a visible light image and an infrared image;

[0007] Inputting the visible light image and the infrared image into a first backbone network and a second backbone network of a target detector respectively;

[0008] Extracting features of the visible light image and the infrared image respectively through the first backbone network and the second backbone network to obtain feature information of the first modality and the second modality, wherein the feature information of the first modality and the second modality includes features of the visible light image and features of the infrared image;

[0009] fusing feature information of the first modality and the second modality through a fusion network module of the object detector to obtain a fusion feature; and

[0010] The classification and positioning information of the input image is obtained according to the fusion features.

[0011] According to a second aspect of the present application, a target detector is provided, characterized in that it includes:

[0012] The first backbone network and the second backbone network are used to respectively extract features of a visible light image and an infrared image in the received input image to obtain feature information of a first modality and a second modality, wherein the feature information of the first modality and the second modality includes features of the visible light image and features of the infrared image;

[0013] A fusion network module, used to fuse the feature information of the first modality and the second modality to obtain a fusion feature; and

[0014] The neck and the head are used to obtain the classification and positioning information of the input image according to the fusion feature.

[0015] According to a third aspect of the present application, there is provided an electronic device, comprising:

[0016] Processor; and

[0017] A memory storing computer instructions, which, when executed by the processor, causes the processor to execute the method described in the first aspect.

[0018] According to a fourth aspect of the present application, a non-transitory computer storage medium is provided, storing a computer program, which, when executed by multiple processors, enables the processors to execute the method described in the first aspect.

[0019] According to the dual-light image detection method and target detector provided by this application, the network model of the target detector is improved, and a feature-level fusion module is introduced to simultaneously annotate the first modality and the second modality, and the dual-light misalignment problem is solved by using the implicit alignment characteristics of feature-level fusion; in addition, decision-level fusion is performed after feature-level fusion, and the accuracy of the detection results can be improved by using two different fusion methods. Moreover, experiments show that when the input scale of the detection model is set to 640×640, the target detector has a good detection speed, which meets the requirements of conventional drone hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present application.

[0021] Figure 1 It is a network structure diagram of a target detector including a Concat fusion module according to the first embodiment of the present application.

[0022] Figure 2 It is a network structure diagram of a target detector including a TransFusion fusion module according to the second embodiment of the present application.

[0023] Figure 3 yes Figure 2 The schematic diagram of the structure of the TransFusion fusion module is shown.

[0024] Figure 4 It is a schematic diagram of image-level fusion, feature-level fusion, and decision-level fusion.

[0025] Figure 5 It is an overall schematic diagram of the fusion of feature level and decision level in a target detector according to one embodiment of the present application.

[0026] Figure 6 This is a schematic diagram of the characteristics of the Concat fusion module (where data annotation is based on infrared as the standard).

[0027] Figure 7 This is a schematic diagram of the features of the TransFusion fusion module.

[0028] Figure 8 is the bi-optical image input to the object detector.

[0029] Fig. 9 This is a detection result diagram after detection by a target detector provided in the present application according to an embodiment of the present application.

[0030] Fig.10 is a flow chart of a dual-light image detection method according to an embodiment of the present application.

[0031] Fig.11 is a flow chart of a dual-light image detection method according to another embodiment of the present application.

[0032] Fig.12 It is a structural diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the examples in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0034] In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0035] Figure 1 is a network structure diagram of an object detector including a Concat fusion module according to the first embodiment of the present application. Figure 1 As shown in FIG. 1 , the target detector includes a backbone network (Backbone), a neck (Neck) and a head (Head), and also includes a Concat fusion module (Contact block). In the target detector network, the backbone network includes two backbone networks (Backbone RGB and Backbone TIR), so that when the target detector network inputs the visible light image (RGB) and the infrared image (TIR) ​​(as shown in FIG. Figure 8 As shown in the figure), the two backbone networks extract the features of visible light images and infrared images respectively.

[0036] exist Figure 1In the embodiment shown, the Concat fusion module fuses the features extracted by the backbone network, focusing on fusing the features of the C2f module. The C2f module is one of the core components of the target detector (such as the Yolov8 target detector), and plays a key role in improving the performance and accuracy of the model. By fusing the feature information of the first modality and the second modality (including visible light images and infrared images), the category information and location information of the first modality and the second modality are learned, such as Figure 6 In the prior art, the original target detector (such as Yolov8 target detector) network subsequently uses the C2f module for feature splicing, which is changed to splicing fusion features in this application. The Concat module in the neck (Neck box) in the prior art is originally connected to C2f, but in this application, Figure 1 As shown, the connection is a Concat fusion module (Concatblock) for subsequent detection.

[0037] exist Figure 1 In the illustrated embodiment, the Concat fusion module (Concatblock) is a local feature fusion network used for feature-level fusion, specifically for fusing features of visible light images and infrared images. Figure 1 The network structure shown mainly includes three stages: unimodal feature extraction, bimodal feature fusion, and detected neck and head. The unimodal feature extraction is first applied to the visible light image and infrared image respectively, which can be expressed by equation (1).

[0038]

[0039] in h, w, c represent the length, width and number of channels of the feature map respectively. and They respectively represent the extracted visible light image features and infrared image features, that is, the feature information of the first modality and the second modality. They represent the visible light feature extraction backbone and the infrared image feature extraction backbone respectively. Represent the visible light input image and infrared input image respectively. Selecting different backbones in the feature extraction stage will correspond to different numbers of fused features.

[0040] Then, the obtained The feature map needs to aggregate different branch features from different spectra in the Concat fusion module (Concatblock) structure, as shown in equation (2):

[0041]

[0042] in, is the fusion feature returned by each layer, ΦconcatFusion The fusion method of the Concat fusion module (Concatblock) fuses the visible light features and infrared features of the corresponding layers. Figure 1 In the illustrated embodiment, the features output by the backbone network include low-level features, high-level features, and correspond to the number of convolutions of the features. The more convolutions (on the Backbone), the higher-level features they are. i represents various Backbone output features, which are feature maps, but their feature dimensions are different. The feature maps of each layer are not necessarily sent to the neck. i mainly refers to the fused features sent to the neck of the target detector. Assume that the input output in the Backbone is [0, 1, 2], and the feature maps of each are 80x80x128, 40x40x256, and 20x20x512. This output feature includes Figure 1 The three features of the SPPF module in (this module is indirectly obtained from the fusion features) are sent to the neck for subsequent processing, and i represents different feature maps for subsequent processing.

[0043] In equation (2), Φ concatFusion It is composed of Concat modules, which use a mixed approach for visible light and infrared, so that the mixed features can focus on both visible light and infrared features. Figure 6 For the car in the upper middle of the left image in , Concat fusion can label the first mode and the second mode at the same time, as expressed in equation (3):

[0044]

[0045] in, It is the fusion feature of each layer. The Concat module is used to concatenate the first modality and the second modality features. Conv represents convolution, and SiLU represents activation function.

[0046] Then, the fusion features The input is sent to the neck of the target detector for multi-scale feature fusion, and then sent to the head for subsequent classification and regression, as expressed in equation (4):

[0047]

[0048] in, Represents classification and positioning information respectively. They represent the neck and head processing of the object detector respectively.

[0049] Figure 5 is a general schematic diagram of feature-level and decision-level fusion in a target detector according to an embodiment of the present application, Figure 1 The embodiment shown corresponds to Figure 5 The section "Yolov8 Concat Fusion" in Figure 5 In the "Yolov8Tranfusion Fusion" section, the Figure 2 The illustrated embodiment is described.

[0050] Figure 2 is a network structure diagram of an object detector including a TransFusion fusion module according to the second embodiment of the present application. Figure 2 As shown, the target detector includes a backbone network (Backbone), a neck (Neck) and a head (Head), and also includes a TransFusion fusion module. In the target detector network, the backbone network includes two backbone networks (Backbone RGB and Backbone TIR), so that when the target detector network inputs a visible light image (RGB) and an infrared image (TIR), the two backbone networks extract features of the visible light image and the infrared image respectively.

[0051] exist Figure 2 In the embodiment shown, the TransFusion fusion module fuses the features extracted by the backbone network, focusing on the features of the C2f module. Through the feature information of the first modality and the second modality, the modality alignment is focused on, the visible light modality is aligned to the infrared modality, and the category knowledge between the first modality and the second modality is learned, such as Figure 7 As shown in Figure 2, the Concat block does not focus on implicit alignment, while the TransFusion block aligns the two images at the feature level.

[0052] Figure 3 yes Figure 2 The schematic diagram of the TransFusion fusion module is shown in Figure 1. Figure 3 The structure shown in the figure explains the fusion process of the TransFusion fusion module on the features extracted by the backbone network. The visible light image features and infrared image features extracted by the backbone network and In the TransFusion fusion module structure, different branch features from different spectra need to be aggregated, as shown in Equation 5:

[0053]

[0054] in, is the fusion feature of each layer, Φ TransFusion It is a fusion method that fuses the visible light image features and infrared image features of the corresponding layers.

[0055] like Figure 3 As shown, Φ TransFusion It consists of AdaptivePool, Flatten, Sampling Module and Concat Module. AdaptivePool is a commonly used operation in deep learning that can convert input feature maps of any size into output feature maps of fixed size. It has two modes: average avg and maximum max. Flatten is used to flatten features.

[0056] The avg and max modes of AdaptivePool are applied to the visible light image features to extract features, and the extracted max and avg features are weighted fused. The flattened features are then downsampled to obtain the visible light features of the nearest neighbor sampling, as shown in equations (6) and (7):

[0057]

[0058] in, is the visible light fusion feature of each layer, F.interpolate nearest It is used to downsample and flatten features. α and β are two parameters used to fuse features. The weights of the two can be learned through training. The initial values ​​of the two can be 0.5, for example.

[0059] Then or at the same time, the avg and max modes of AdaptivePool are applied to the infrared image features to extract features, and the extracted max and avg features are weighted fused. After that, the flattened features are downsampled to obtain the visible light features of the nearest neighbor sampling, as shown in equations (8) and (9):

[0060]

[0061] in, is the infrared fusion feature of each layer, F.interpolate nearest It is used to downsample and flatten features. γ and δ are two parameters used to fuse features. The weights of the two can be learned through training. The initial values ​​of the two can be 0.5, for example.

[0062] Then, if Figure 3 As shown, the Concat module is needed to fuse the obtained visible light fusion features and infrared fusion features, and then perform convolution operation to obtain the final fusion features, which can be expressed by equation 10:

[0063]

[0064] in, represents fusion features, and Conv represents convolution module.

[0065] Finally, the fusion features The input is sent to the neck of the target detector for multi-scale feature fusion, and then sent to the head for subsequent classification and regression. Figure 1 Different from the feature-level fusion of , the fusion result focuses on the detection of infrared modality, as shown in equation (11):

[0066]

[0067] in, Represents classification and positioning information respectively. Represents the neck and head processing of an object detector.

[0068] According to some embodiments, the image needs to be preprocessed before it is input into the target detector. The preprocessing performed includes a series of data enhancements, which are finally converted into tensors. The input image is preprocessed. According to a specific embodiment, the input image size is 640×640. Applied data enhancements include rotation data enhancement (random rotation of 90 degrees), multi-scale training (ranging from 0.5 to 1.5), Mosaic data enhancement, etc. These data enhancements are synchronized on visible light and infrared images, that is, the data enhancement operations of the two are completely consistent in the first mode and the second mode. Finally, the visible light and infrared images are converted into tensors. The use of data enhancement processing can further improve the accuracy of target detection.

[0069] The visible light image and infrared image are input into different backbone networks, and the network model of the target detector is trained based on the two fusion network modules. Figure 1 and Figure 2 The obtained classification and positioning information are merged, for example, using WBF (weighted boxes fusion) to fuse the results. Figure 5 The "WBF" shown in FIG. 4 shows how the fusion is performed at the decision level. In addition, other merging methods may be used, such as NMS (non-maximum suppression) and softnms, etc., and this application does not impose any restrictions on this.

[0070] According to some embodiments, the target detector used in the present application may be any suitable target detection algorithm model, such as Yolov8, etc., and the present application does not impose any limitation on this.

[0071] Figure 4 Three fusion methods are shown, namely image-level fusion, feature-level fusion and decision-level fusion. Among them, traditional image-level fusion and decision-level fusion cannot effectively deal with non-aligned images, such as Figure 6The car in the upper middle of the left picture is not synchronized in the first mode and the second mode. Traditional image-level fusion and decision-level fusion are difficult to effectively process non-aligned images. This application chooses to apply feature-level fusion (such as Figure 1 and Figure 2 As shown in the figure, we use the visible light image and infrared image to input different backbone networks, use two fusion methods for training, and use the trained network to detect the test data. To further improve the accuracy, we use the target detector under two fusion methods (feature-level fusion and decision-level fusion) for detection, and finally merge the results through weighted box fusion (WBF). The final detection effect is shown in the figure. Fig. 9 shown.

[0072] Based on the above description of the detector, according to one aspect of the present application, a dual-light image detection method is provided. Fig.10 FIG. 1 is a flow chart of a dual light image detection method according to an embodiment of the present application. Fig.10 As shown, the method comprises the following steps:

[0073] Step S1001, receiving an input image, wherein the input image includes a visible light image and an infrared image;

[0074] Step S1002, inputting the visible light image and the infrared image into a first backbone network and a second backbone network of a target detector respectively;

[0075] Step S1003, extracting features of the visible light image and the infrared image respectively through the first backbone network and the second backbone network to obtain feature information of the first modality and the second modality, wherein the feature information of the first modality and the second modality includes features of the visible light image and features of the infrared image;

[0076] Step S1004, fusing the feature information of the first modality and the second modality through the fusion network module of the object detector to obtain a fusion feature; and

[0077] Step S1005: obtaining classification and positioning information of the input image according to the fusion features.

[0078] As mentioned above, Figure 1 and Figure 2In the network structure of the target detector shown, images are received, including visible light images and infrared images. Then, the first backbone network and the second backbone network of the target detector respectively extract features of the visible light image and the infrared image to obtain feature information of the first modality and the second modality, and the feature information of the first modality and the second modality is fused through the fusion network module of the target detector (for example, the Concat fusion module and the TransFusion fusion module), and the fused features are processed to obtain classification and positioning information of the input image.

[0079] According to some optional embodiments, step S1004 may include:

[0080] fusing feature information of the first modality and the second modality through the first fusion network module to obtain a first fusion feature; and

[0081] The feature information of the first modality and the second modality is fused through the second fusion network module to obtain a second fusion feature.

[0082] exist Figure 1 and Figure 2 In the illustrated embodiment, the feature information of the first modality and the second modality are fused by the Concat fusion module and the TransFusion fusion module, respectively, to obtain the first fusion feature and the second fusion feature. According to a specific embodiment, for the Concat fusion module, the feature information of the first modality and the second modality are processed by the Concat module, the convolution module and the activation function module, respectively, to obtain the first fusion feature. According to another specific embodiment, for the TransFusion fusion module, the visible light feature information and the infrared feature information in the feature information of the first modality and the second modality are respectively subjected to adaptive pooling to obtain the visible light adaptive pooling result and the infrared adaptive pooling result; the visible light adaptive pooling result and the infrared adaptive pooling result are respectively flattened to obtain the visible light flattening result and the infrared flattening result; the visible light flattening result and the infrared flattening result are respectively downsampled to obtain the visible light fusion feature and the infrared fusion feature; then, the visible light fusion feature and the infrared fusion feature are processed by the Concat module and the convolution module in turn to obtain the second fusion feature.

[0083] According to some optional embodiments, step S1005 may include:

[0084] The first fusion feature and the second fusion feature are processed by the neck and the head of the target detector respectively to obtain first classification and positioning information and second classification and positioning information of the input image.

[0085] According to some optional embodiments, step S1005 may further include:

[0086] The first classification and positioning information and the second classification and positioning information are combined to obtain the classification and positioning information of the input image.

[0087] like Figure 1 and Figure 2 In the embodiment shown, the first fusion feature and the second fusion feature are respectively input into the neck of the target detector for multi-scale feature fusion, and then sent to the head to complete subsequent classification and regression, so as to obtain the first classification and positioning information and the second classification and positioning information of the input image.

[0088] After obtaining the first classification and positioning information and the second classification and positioning information, such as Figure 5 As shown in the figure, after the training is completed, the network model is used in the inference phase. Figure 1 and Figure 2 The first classification and positioning information and the second classification and positioning information obtained are merged, for example, using WBF (weighted boxes fusion) to fuse the results to obtain a detection result.

[0089] Based on the above description of the detector, according to another aspect of the present application, a dual light image detection method is provided, such as Fig.11 As shown. Fig.10 compared to, Fig.11 Steps S1101, S1103 to S1106 of the method shown in FIG. Fig.10 The steps S1001 to S1005 are the same except that: Fig.11 The method shown also includes:

[0090] Step S1102, preprocessing the input image, wherein the preprocessing includes data enhancement processing.

[0091] According to some embodiments, the image needs to be preprocessed before it is input into the target detector. The preprocessing performed includes a series of data enhancements, which are finally converted into tensors. The input image is preprocessed. According to a specific embodiment, the input image size is 640×640. Applied data enhancements include rotation data enhancement (random rotation of 90 degrees), multi-scale training (ranging from 0.5 to 1.5), Mosaic data enhancement, etc. These data enhancements are synchronized on visible light and infrared images, that is, the data enhancement operations of the two are completely consistent in the first mode and the second mode. Finally, the visible light and infrared images are converted into tensors. Then, the preprocessed image is input into the network structure of the target detector.

[0092] The target detection method provided in this application is mainly aimed at dual-light target detection scenarios, and mainly solves the problem of dual-light detection data misalignment and detection accuracy under aerial photography data. The solution of this application improves the network structure of the existing target detector, while maintaining the accuracy of the target detector itself, it solves a series of challenges faced by dual-light scenarios; it adopts two feature-level fusions, feature-level fusion and decision-level fusion, and uses the characteristics of implicit alignment of feature-level fusion to solve the dual-light misalignment problem. After feature-level fusion, decision-level fusion is performed, and two different fusion methods are used to generate accurate detection results; moreover, experiments show that when the input scale of the detection model is set to 640×640, the target detector has a good detection speed, which meets the requirements of conventional drone hardware platforms.

[0093] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0095] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical connection or other forms.

[0096] See also Fig.12 , Fig.12 An electronic device is provided, comprising a processor and a memory. The memory stores computer instructions, and when the computer instructions are executed by the processor, the processor executes the computer instructions to achieve the following Fig.10 and Fig.11 The method and refinement scheme shown.

[0097] It should be understood that the above device embodiments are only illustrative, and the device disclosed in the present invention can also be implemented in other ways. For example, the division of units / modules described in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0098] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present invention may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0099] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not otherwise specified, the processor or chip may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not otherwise specified, the on-chip cache, the off-chip memory, and the memory may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0100] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer electronic device (which can be a personal computer, a server or a network electronic device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0101] The present application also provides a non-transitory computer storage medium storing a computer program. When the computer program is executed by multiple processors, the processors execute the following Fig.10 and Fig.11 The method and refinement scheme shown.

[0102] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. A dual light image detection method, characterized in that: include: receiving an input image, wherein the input image includes a visible light image and an infrared image; Inputting the visible light image and the infrared image into a first backbone network and a second backbone network of a target detector respectively; Extracting features of the visible light image and the infrared image respectively through the first backbone network and the second backbone network to obtain feature information of the first modality and the second modality, wherein the feature information of the first modality and the second modality includes features of the visible light image and features of the infrared image; fusing feature information of the first modality and the second modality through a fusion network module of the object detector to obtain a fusion feature; and The classification and positioning information of the input image is obtained according to the fusion features.

2. The method according to claim 1, characterized in that The fusion network module includes a first fusion network module and a second fusion network module, and the fusion network module of the target detector fuses the feature information of the first modality and the second modality to obtain a fusion feature, including: fusing feature information of the first modality and the second modality through the first fusion network module to obtain a first fusion feature; and The feature information of the first modality and the second modality is fused through the second fusion network module to obtain a second fusion feature.

3. The method according to claim 2, characterized in that The fusing the feature information of the first modality and the second modality through the first fusion network module to obtain a first fusion feature includes: The feature information of the first modality and the second modality are processed by a Concat module, a convolution module and an activation function module respectively to obtain the first fusion feature.

4. The method according to claim 2, characterized in that The fusing the feature information of the first modality and the second modality through the second fusion network module to obtain a second fusion feature includes: Adaptively pooling the visible light feature information and the infrared feature information in the feature information of the first modality and the second modality to obtain a visible light adaptive pooling result and an infrared adaptive pooling result; Flattening the visible light adaptive pooling result and the infrared adaptive pooling result respectively to obtain a visible light flattening result and an infrared flattening result; Downsampling the visible light flattening result and the infrared flattening result respectively to obtain visible light fusion features and infrared fusion features; and The visible light fusion feature and the infrared fusion feature are processed by a Concat module and a convolution module in sequence to obtain the second fusion feature.

5. The method according to claim 2, characterized in that The obtaining the classification and positioning information of the input image according to the fusion feature includes: The first fusion feature and the second fusion feature are processed by the neck and the head of the target detector respectively to obtain first classification and positioning information and second classification and positioning information of the input image.

6. The method according to claim 5, characterized in that The obtaining the classification and positioning information of the input image according to the fusion feature further includes: The first classification and positioning information and the second classification and positioning information are combined to obtain the classification and positioning information of the input image.

7. The method according to claim 1, characterized in that Before inputting the visible light image and the infrared image into the first backbone network and the second backbone network of the target detector respectively, the method further includes: The input image is preprocessed, wherein the preprocessing includes data enhancement processing.

8. The method according to claim 1, characterized in that The object detector includes a Yolov8 detector.

9. A target detector, characterized in that: include: The first backbone network and the second backbone network are used to respectively extract features of a visible light image and an infrared image in the received input image to obtain feature information of a first modality and a second modality, wherein the feature information of the first modality and the second modality includes features of the visible light image and features of the infrared image; A fusion network module, used to fuse the feature information of the first modality and the second modality to obtain a fusion feature; and The neck and the head are used to obtain the classification and positioning information of the input image according to the fusion feature.

10. The detector according to claim 9, characterized in that The fusion network module includes: A first fusion network module, configured to fuse feature information of the first modality and the second modality to obtain a first fusion feature; and The second fusion network module is used to fuse the feature information of the first modality and the second modality to obtain a second fusion feature.

11. The detector according to claim 9, characterized in that The neck and the head are used for: The first fusion feature and the second fusion feature are processed respectively to obtain first classification and positioning information and second classification and positioning information of the input image.

12. The detector according to claim 11, characterized in that The neck and the head are also used for: The first classification and positioning information and the second classification and positioning information are combined to obtain the classification and positioning information of the input image.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program in the memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images