Cross-domain information fusion target detection method and device

Abstract

The invention provides a cross-domain information fusion target detection method and device, and the device is characterized in that the device comprises a double-flow target detection model which comprises a feature extraction module which carries out the feature extraction of a registered mode A image to obtain a high-level semantic feature A, and carries out the feature extraction of a registered mode B image to obtain a high-level semantic feature B; the multi-scale enhancement module is used for performing multi-scale enhancement on a feature pair formed by the high-level semantic feature A and the high-level semantic feature B to obtain an enhanced input feature pair; the fusion module is used for carrying out space-level multi-modal interaction and channel-level multi-modal interaction on the enhanced input feature pair to obtain a multi-modal feature; and the target detection module is used for generating a detection frame according to the multi-modal features and taking the confidence coefficient of the corresponding category as a detection result. In a word, the method can realize accurate detection of various targets in a complex scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a target detection method and device for cross-domain information fusion. Background Art

[0002] As a popular research direction in the field of computer vision, object detection has been widely used in many fields such as intelligent driving, medical diagnosis and security monitoring, greatly promoting the development of productivity. According to candidate prior information, current object detection methods can be divided into three types: anchor-based object detection method, anchor-based object detection method and query-based object detection method.

[0003] The anchor-based object detection method is usually divided into two steps. The first step is to generate a series of regions of interest, and the second step is to fine-tune the location of the regions of interest and determine their categories. The anchor-based object detection method is an improvement on the anchor-based method with more manual components. The query-based object detection method is a more flexible detection method that does not rely on anchors and anchor boxes. It searches the entire image for the target you want to focus on through a predefined fixed number of queries.

[0004] However, most of the above-mentioned target detection methods are based on visible light images, which makes it difficult to ensure the stability of the detection system in various practical application scenarios such as heavy fog, heavy rain, and night. Especially in night scenes, visible light images cannot provide sufficient foreground target information. Although technologies such as dark light enhancement are constantly developing, detection based only on single light images is difficult to meet the application requirements in replication scenarios. Therefore, researchers have introduced multimodal fusion methods of images of other modalities to alleviate this problem.

[0005] However, due to the huge semantic information gap between different modalities, the existing multimodal fusion methods have not fully explored the correlation between different modalities, so that the target detection results of the existing multimodal fusion methods still have a lot of room for improvement. Summary of the invention

[0006] The present invention is made to solve the above-mentioned problem, and its purpose is to provide a target detection method and device with cross-domain information fusion.

[0007] The present invention provides a target detection method for cross-domain information fusion, which is used for obtaining a detection result of a target according to a modality A image and a modality B image corresponding to the target, and has the following characteristics, including the following steps: step S1, registering the modality A image and the modality B image to obtain a registered modality A image and a modality B image; step S2, inputting the registered modality A image and the registered modality B image into a dual-stream target detection model to obtain a detection result, wherein the dual-stream target detection model includes: a feature extraction module, performing feature extraction on the registered modality A image to obtain a high-level semantic feature A, and performing feature extraction on the registered modality B image to obtain a high-level semantic feature B; a multi-scale enhancement module, performing multi-scale enhancement on a feature pair consisting of the high-level semantic feature A and the high-level semantic feature B to obtain an enhanced input feature pair; a fusion module, performing spatial-level multi-modal interaction and channel-level multi-modal interaction on the enhanced input feature pair to obtain a multi-modal feature; a target detection module, generating a detection frame and a corresponding category confidence as a detection result according to the multi-modal feature.

[0008] The target detection method of cross-domain information fusion provided by the present invention may also have the following characteristics: wherein the feature extraction module includes The calculation expressions of the backbone network, high-level semantic features A and high-level semantic features B are: , , where is the high-level semantic feature A, is the modality A image, is the high-level semantic feature B, is the mode B image.

[0009] The target detection method of cross-domain information fusion provided by the present invention may also have the following characteristics: in the multi-scale enhancement module, the feature pairs are upsampled to multiple different multiples in width and height respectively to obtain corresponding feature pairs of multiple different scales, and all feature pairs of different scales are expanded into sequences in width and height and spliced ​​to obtain enhanced input feature pairs.

[0010] In the target detection method of cross-domain information fusion provided by the present invention, it can also have the following characteristics: wherein, the fusion module includes multiple high-order collaborative interaction units connected in sequence, the high-order collaborative interaction unit includes a spatial-level multimodal interaction sub-unit and a channel-level multimodal interaction sub-unit, the spatial-level multimodal interaction sub-unit performs a compromise split on the input in the channel dimension, and fuses the split results in the spatial dimension through a deformable attention mechanism and a residual connection to obtain a corresponding output, the channel-level multimodal interaction sub-unit performs channel dimension splicing, sequence dimension splitting, three-dimensional reshaping and channel-level attention processing on the input in sequence to obtain a corresponding output, in the high-order collaborative interaction unit, the output of the spatial-level multimodal interaction sub-unit of the high-order collaborative interaction unit is used as the input of the channel-level multimodal interaction sub-unit of the high-order collaborative interaction unit, and the output of the channel-level multimodal interaction sub-unit is used as the input of the spatial-level multimodal interaction sub-unit of the next high-order collaborative interaction unit, and the input of the spatial-level multimodal interaction sub-unit of the first high-order collaborative interaction unit is an enhanced input feature pair.

[0011] The target detection method of cross-domain information fusion provided by the present invention may also have the following characteristics: wherein the spatial multimodal interaction subunit inputs Split into feature pairs of the same shape and feature pairs , spatial level multimodal interaction subunit pair feature pair and feature pairs Fusion in the spatial dimension gives the output and The calculation expression is: , , , where is the number of attention heads in the deformable attention mechanism, and They are the fully connected layers, To increase the number of different scales corresponding to features in the input feature pair, For the The number of sampling points for an attention head, For the The attention weights of the attention heads, For the corresponding The sampling point is The attention head is based on the offset Updated sampling points.

[0012] The target detection method of cross-domain information fusion provided by the present invention may also have the following characteristics: wherein the channel-level multimodal interaction subunit performs input and Generate Output The calculation expression is: , , , , , , where For channel-level splicing operations, For the split operation, For reshaping operations, To increase the number of different scales corresponding to features in the input feature pair, feature Height, feature The width of feature Center coordinate point The corresponding value, are learnable parameters.

[0013] In the target detection method of cross-domain information fusion provided by the present invention, it can also have the following characteristics: wherein, the construction and training process of the dual-stream target detection model includes the following steps: a training data construction step, constructing a training data set according to existing target images of multiple different modalities; a model construction step, constructing a dual-stream target detection model; a model training step, training the dual-stream target detection model according to the training data set to obtain a trained dual-stream target detection model, in the model training step, the loss function of training the dual-stream target detection model includes classification loss and target positioning loss, and the calculation expression of the classification loss is: , where is the regulating factor, The predicted labels generated by the two-stream target detection model based on the image, is the true classification label corresponding to the image, The calculation result of classification loss is as follows: , where is the prediction box, For the prediction box The corresponding annotation box, Predicting boxes for packages and callout box The minimum bounding box of For the prediction box and callout box The intersection-combination ratio between Calculate the result for target localization loss.

[0014] The target detection method of cross-domain information fusion provided by the present invention may also have the following features: wherein, in the training data construction step, at least two images of different modalities are collected for the existing target at the same shooting position in each scene as the existing target images, and all existing target images corresponding to the same shooting position and the same existing target are paired in pairs to construct image pairs, and the two existing target images in the image pair are respectively labeled with the target category and position information of the corresponding existing target, and the two target images are aligned to eliminate misalignment.

[0015] The present invention also provides a target detection device for cross-domain information fusion, which is used to obtain the detection result of the target according to the modal A image and the modal B image corresponding to the target, and has the following characteristics, including: a registration unit, which is used to align the modal A image and the modal B image to obtain the registered modal A image and the modal B image; a detection unit, which includes a dual-stream target detection model, which is used to input the registered modal A image and the registered modal B image into the dual-stream target detection model to obtain the detection result, wherein the dual-stream target detection model includes: a feature extraction module, which performs feature extraction on the registered modal A image to obtain the high-level semantic feature A, and performs feature extraction on the registered modal B image to obtain the high-level semantic feature B; a multi-scale enhancement module, which performs multi-scale enhancement on the feature pair composed of the high-level semantic feature A and the high-level semantic feature B to obtain the enhanced input feature pair; a fusion module, which performs spatial-level multi-modal interaction and channel-level multi-modal interaction on the enhanced input feature pair to obtain multi-modal features; a target detection module, which includes a detection head, which generates a detection frame and a corresponding category confidence as a detection result according to the multi-modal features.

[0016] According to the target detection method and device of cross-domain information fusion involved in the present invention, on the one hand, the complementary feature information of the two modalities at the spatial level is adaptively captured by using the deformable attention mechanism through the spatial-level multimodal interaction subunit; on the other hand, the channel-level attention block is used by the channel-level multimodal interaction subunit to model the channel interaction between the two different modalities, thereby realizing complementary information fusion and redundant information filtering between multimodalities. Therefore, the target detection method and device of cross-domain information fusion of the present invention can realize accurate detection of various targets in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram of a target detection device in an embodiment of the present invention.

[0018] Figure 2 is a block diagram of a dual-stream target detection model in an embodiment of the present invention.

[0019] Figure 3 It is a flowchart of the construction and training of a dual-stream target detection model in an embodiment of the present invention.

[0020] Figure 4 It is a flowchart of a target detection method for cross-domain information fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the target detection method and device for cross-domain information fusion of the present invention.

[0022] In this embodiment, a target detection device for cross-domain information fusion is provided, hereinafter referred to as the target detection device, which is used to obtain a detection result of the target according to the modality A image and the modality B image corresponding to the target. In this embodiment, modality A and modality B are different modalities, and visible light images, infrared imaging images, lidar imaging images, and synthetic aperture radar imaging images are all images of different modalities.

[0023] Figure 1 is a block diagram of a target detection device in an embodiment of the present invention.

[0024] like Figure 1 As shown, the target detection device 100 includes a registration unit 11 and a detection unit 12 .

[0025] The registration unit 11 is used to register the modality A image and the modality B image to obtain the registered modality A image and modality B image.

[0026] Due to the difference in imaging principles, even if two images of different modalities with the same target height are continuously taken at the same shooting position, there is still a common misalignment between the two images to varying degrees. If there is an offset or deformation between the image pairs, artifacts will inevitably appear in the subsequent fusion results. To address this problem, an adaptive symmetric registration algorithm is used in this embodiment to register the target images. The following describes the registration process by taking the infrared image and the visible light image as the modal A image and the modal B image, respectively: The infrared image is input into a trained infrared-to-visible light aligner to obtain a registered infrared image, and the visible light image is input into a trained visible light-to-infrared aligner to obtain a registered visible light image.

[0027] When training the infrared to visible light aligner, the infrared image is affine transformed to obtain a transformation matrix. Then, the visible light image and the transformed infrared image are used as the input of the aligner, and the parameters of the transformation matrix are used as the annotations that the aligner wants to predict, so as to learn the infrared image before the transformation. The purpose is that when the input visible light image and the infrared image are misaligned, the infrared image can be corrected according to the aligner to obtain an infrared image that accurately corresponds to the position of the visible light image.

[0028] When training a visible-to-infrared aligner, an affine transformation is performed on the visible light image to obtain a transformation matrix. Then, the infrared image and the transformed visible light image are used as inputs to the aligner, and the parameters of the transformation matrix are used as the annotations that the aligner wants to predict, so as to learn the visible light before the transformation. The purpose is that when the input infrared image and the visible light image are misaligned, the visible light image can be corrected according to the aligner to obtain a visible light image that accurately corresponds to the position of the infrared image.

[0029] The detection unit 12 includes a dual-stream target detection model, which is used to input the registered modality A image and the registered modality B image into the dual-stream target detection model to obtain a detection result.

[0030] Figure 2 is a block diagram of a dual-stream target detection model in an embodiment of the present invention.

[0031] like Figure 2 As shown, the dual-stream target detection model 200 includes a feature extraction module 21, a multi-scale enhancement module 22, a fusion module 23 and a target detection module 24.

[0032] The feature extraction module 21 includes two parallel The backbone network extracts features from the registered modality A image to obtain high-level semantic features A, and extracts features from the registered modality B image to obtain high-level semantic features B. In this embodiment, the size of the input image of the feature extraction module 21 is 1024. The feature extraction module 21 first processes the input image through a 16-fold downsampling layer to obtain a feature map of size 64×64, and then obtains the backbone features of different modalities, i.e., high-level semantic features, through 24 ViT blocks. Among them, the random dropout rate is 0.3, and the hidden layer dimension is set to 1024.

[0033] Among them, the calculation expressions of high-level semantic features A and B are: , , In the formula is the high-level semantic feature A, is the modality A image, is the high-level semantic feature B, is the modality B image. Among them, is the number of feature channels, and Indicates the height and width of the last layer.

[0034] The multi-scale enhancement module 22 performs multi-scale enhancement on the feature pair consisting of the high-level semantic feature A and the high-level semantic feature B to obtain an enhanced input feature pair.

[0035] Among them, the multi-scale enhancement module 22 samples the feature pairs in width and height by multiple different multiples respectively to obtain corresponding feature pairs of multiple different scales, expands all feature pairs of different scales into sequences in width and height and splices them to obtain enhanced input feature pairs.

[0036] In this embodiment, the multiples include 2, 4, 8 and 16 times. For the multiple multi-scale features obtained by upsampling, they are further enhanced through several convolutional layers to obtain enhanced input feature pairs. Among them, the number of output channels is set to 256, and layer normalization is used for regularization and GeLU activation function is selected for nonlinear transformation. By feature pair Composition, among which, is the sequence dimension after concatenation of multi-scale features, When the feature pair is the feature pair input to the multi-scale enhancement module 22, When , the feature pairs are formed by multi-scale features obtained by upsampling by 2 times, 4 times, 8 times, and 16 times.

[0037] The fusion module 23 performs spatial-level multimodal interaction and channel-level multimodal interaction on the enhanced input feature pairs to obtain multimodal features.

[0038] The fusion module 23 includes a plurality of sequentially connected high-order collaborative interaction units 231. In this embodiment, three high-order collaborative interaction units 231 are sequentially connected. The high-order collaborative interaction unit 231 includes a space-level multimodal interaction subunit 2311 and a channel-level multimodal interaction subunit 2312.

[0039] In the high-order collaborative interaction unit 231, the output of the spatial-level multimodal interaction subunit 2311 of the high-order collaborative interaction unit 231 is used as the input of the channel-level multimodal interaction subunit 2312 of the high-order collaborative interaction unit 231, and the output of the channel-level multimodal interaction subunit 2312 is used as the input of the spatial-level multimodal interaction subunit 2311 of the next high-order collaborative interaction unit 231. The input of the spatial-level multimodal interaction subunit 2311 of the first high-order collaborative interaction unit 231 is the enhanced input feature pair.

[0040] The spatial-level multimodal interaction subunit 2311 splits the input in the channel dimension, and fuses the split results in the spatial dimension through a deformable attention mechanism and residual connection to obtain the corresponding output.

[0041] Among them, the spatial multimodal interaction subunit 2311 inputs Compromise split into feature pairs of the same shape and feature pairs In this embodiment, the compromise is split into The dimension N will be input The first N / 2 dimensions are used as feature pairs , and enter The N / 2 dimensions in the middle and back are used as feature pairs .

[0042] Spatial multimodal interaction subunit 2311 pairs of features and feature pairs Fusion in the spatial dimension gives the output and The calculation expression is: , , , In the formula is the number of attention heads in the deformable attention mechanism, and They are the fully connected layers, To increase the number of different scales corresponding to features in the input feature pair, For the The number of sampling points for an attention head, For the The attention weights of the attention heads, For the corresponding The sampling point is The attention head is based on the offset The updated sampling point. is 5, which corresponds to the feature pair .

[0043] Existing spatial-level fusion strategies usually use Transformer for fusion. However, directly sampling Transformer for fusion of long sequence features after multi-scale feature splicing will bring heavy memory burden and high time complexity, and will introduce more redundant noise impurities of failure modes. The deformable attention mechanism adopted by the spatial-level multimodal interaction subunit 2311 can effectively avoid the above problems.

[0044] The channel-level multimodal interaction subunit 2312 performs channel dimension splicing, sequence dimension splitting, three-dimensional reshaping and channel-level attention processing on the input in sequence to obtain the corresponding output.

[0045] Among them, the channel-level multimodal interaction subunit 2312 pairs of input and Generate Output The calculation expression is: , , , , , , In the formula For channel-level splicing operations, For the split operation, For reshaping operations, To increase the number of different scales corresponding to features in the input feature pair, feature Height, feature The width of feature Center coordinate point The corresponding value, are learnable parameters.

[0046] In this embodiment, the collaborative correlation between modality A and modality B is gradually captured through iterative interactions of multiple spatial-level multimodal interaction subunits 2311 and channel-level multimodal interaction subunits 2312. Specifically, the spatial-level multimodal interaction subunit 2311 integrates spatial fine-grained complementary information through a deformable attention mechanism. The channel-level multimodal interaction subunit 2312 uses global first-order statistics to distinguish the mutual dependence between modality A and modality B.

[0047] The target detection module 24 includes the encoder and decoder layers of Co-DETR, and generates a detection box and a corresponding category confidence as a detection result according to the multimodal features. In this embodiment, the multimodal features are extracted by Co-DETR, and the confidence of the detection box and the category are obtained by two multi-layer perceptrons respectively.

[0048] Figure 3 It is a flowchart of the construction and training of a dual-stream target detection model in an embodiment of the present invention.

[0049] like Figure 3 As shown, the construction and training process of the dual-stream target detection model 200 includes the following steps: Step T1 is a training data construction step, which constructs a training data set based on existing target images of multiple different modalities.

[0050] In each scenario, at least two images of different modalities are collected at the same shooting position for the existing target as the existing target image. All existing target images corresponding to the same shooting position and the same existing target are paired to form an image pair. The two existing target images in the image pair are respectively labeled with the target category and position information of the corresponding existing target, and the two target images are registered to eliminate misalignment.

[0051] In this embodiment, the shooting scenes of the existing target include but are not limited to complex scenes such as occlusion, dim light, foggy day, rainy day, etc. The target category and location information of the existing target are annotated by Labelme annotation software, and the annotated data is converted into the COCO dataset format type.

[0052] Step T2 is a model building step, which builds a dual-stream target detection model 200.

[0053] In this embodiment, step T1 is executed first and then step T2. In other embodiments, step T2 may be executed first and then step T1, or step T1 and step T2 may be executed simultaneously as needed.

[0054] Step T3 is a model training step, in which the dual-stream object detection model 200 is trained according to the training data set to obtain a trained dual-stream object detection model 200.

[0055] The loss function for training the dual-stream target detection model 200 includes classification loss and target positioning loss.

[0056] The calculation expression of classification loss is: , In the formula is the regulating factor, The predicted labels generated by the two-stream target detection model based on the image, is the true classification label corresponding to the image, Calculate the result for the classification loss.

[0057] The calculation expression of target positioning loss is: , In the formula is the prediction box, For the prediction box The corresponding annotation box, Predicting boxes for packages and callout box The minimum bounding box of For the prediction box and callout box The intersection-combination ratio between Calculate the result for target localization loss.

[0058] The following describes the process of a target detection method that uses a target detection method device 100 to perform cross-domain information fusion in conjunction with the accompanying drawings.

[0059] Figure 4 It is a flowchart of a target detection method for cross-domain information fusion in an embodiment of the present invention.

[0060] like Figure 4 As shown, the target detection method of cross-domain information fusion includes the following steps: Step S1 : registering the modality A image and the modality B image by using the registration unit 11 to obtain registered modality A image and modality B image.

[0061] In step S2, the detection unit 12 is used to input the registered modality A image and the registered modality B image into a dual-stream target detection model to obtain a detection result.

[0062] Functions and Effects of the Embodiments According to the target detection method and device for cross-domain information fusion involved in this embodiment, on the one hand, the spatial-level multimodal interaction subunit uses a deformable attention mechanism to adaptively capture the complementary feature information of the two modalities at the spatial level; on the other hand, the channel-level multimodal interaction subunit uses a channel-level attention block to model the channel interaction between the two different modalities, thereby realizing complementary information fusion and redundant information filtering between multimodalities. In short, this method can realize accurate detection of various targets in complex scenes.

[0063] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A target detection method using cross-domain information fusion, for obtaining a detection result of a target according to a modality A image and a modality B image corresponding to the target, characterized in that: The following steps are involved: Step S1, registering the modality A image and the modality B image to obtain a registered modality A image and modality B image; Step S2, inputting the registered modality A image and the registered modality B image into a dual-stream object detection model to obtain the detection result, Wherein, the dual-stream target detection model includes: A feature extraction module performs feature extraction on the registered modality A image to obtain a high-level semantic feature A, and performs feature extraction on the registered modality B image to obtain a high-level semantic feature B; A multi-scale enhancement module performs multi-scale enhancement on the feature pair consisting of the high-level semantic feature A and the high-level semantic feature B to obtain an enhanced input feature pair; A fusion module performs spatial-level multimodal interaction and channel-level multimodal interaction on the enhanced input feature pair to obtain multimodal features; The target detection module generates a detection frame and a confidence level of a corresponding category as the detection result according to the multimodal feature.

2. The target detection method based on cross-domain information fusion according to claim 1 is characterized in that: in, The feature extraction module comprises Backbone network, The calculation expressions of the high-level semantic feature A and the high-level semantic feature B are: ， ， In the formula is the high-level semantic feature A, is the modality A image, is the high-level semantic feature B, is the mode B image.

3. The target detection method of cross-domain information fusion according to claim 1 is characterized in that: in, In the multi-scale enhancement module, the feature pairs are upsampled by multiple different multiples in width and height to obtain corresponding feature pairs of multiple different scales, and all the feature pairs of different scales are expanded into sequences in width and height and spliced ​​to obtain the enhanced input feature pairs.

4. The target detection method of cross-domain information fusion according to claim 1 is characterized in that: in, The fusion module includes a plurality of sequentially connected high-order collaborative interaction units, The high-order collaborative interaction unit includes a space-level multimodal interaction subunit and a channel-level multimodal interaction subunit. The spatial multimodal interaction subunit splits the input in the channel dimension, and fuses the split results in the spatial dimension through a deformable attention mechanism and residual connection to obtain the corresponding output. The channel-level multimodal interaction subunit sequentially performs channel dimension splicing, sequence dimension splitting, three-dimensional reshaping and channel-level attention processing on the input to obtain the corresponding output. In the high-order collaborative interaction unit, the output of the spatial-level multimodal interaction subunit of the high-order collaborative interaction unit is used as the input of the channel-level multimodal interaction subunit of the high-order collaborative interaction unit, and the output of the channel-level multimodal interaction subunit is used as the input of the spatial-level multimodal interaction subunit of the next high-order collaborative interaction unit. The input of the spatial-level multimodal interaction subunit of the first high-order collaborative interaction unit is the enhanced input feature pair.

5. The target detection method of cross-domain information fusion according to claim 4 is characterized in that: in, The spatial multimodal interaction subunit inputs Compromise split into feature pairs of the same shape and feature pairs , The spatial multimodal interaction subunit is used to interact with the feature pair and the characteristic pair Fusion in the spatial dimension gives the output and The calculation expression is: ， ， ， In the formula is the number of attention heads in the deformable attention mechanism, and They are the fully connected layers, To increase the number of different scales corresponding to features in the input feature pair, For the The number of sampling points for an attention head, For the The attention weights of the attention heads, For the corresponding The sampling point is The attention head is based on the offset Updated sampling points.

6. The target detection method of cross-domain information fusion according to claim 4 is characterized in that: in, The channel-level multimodal interaction subunit is input and Generate Output The calculation expression is: ， ， ， ， ， ， In the formula For channel-level splicing operations, For the split operation, For reshaping operations, To increase the number of different scales corresponding to features in the input feature pair, Features Height, Features The width of Features Center coordinate point The corresponding value, are learnable parameters.

7. The target detection method of cross-domain information fusion according to claim 1, Features: The construction and training process of the dual-stream target detection model includes the following steps: A training data construction step, constructing a training data set based on existing target images of multiple different modalities; Model building step: building a two-stream target detection model; The model training step is to train the dual-stream object detection model according to the training data set to obtain a trained dual-stream object detection model. In the model training step, the loss function of training the dual-stream target detection model includes classification loss and target positioning loss. The calculation expression of the classification loss is: ， In the formula is the regulating factor, The predicted labels generated by the two-stream target detection model based on the image, is the true classification label corresponding to the image, is the classification loss calculation result, The calculation expression of the target positioning loss is: ， In the formula is the prediction box, For the prediction box The corresponding annotation box, Predicting boxes for packages and callout box The minimum bounding box of For the prediction box and callout box The intersection-combination ratio between Calculate the result for target localization loss.

8. The target detection method of cross-domain information fusion according to claim 7 is characterized in that: in, In the training data construction step, at least two images of different modalities are collected from the existing target at the same shooting position in each scene as the existing target image, Pair all the existing target images corresponding to the same shooting position and the same existing target in pairs to construct image pairs, The two existing target images in the image pair are respectively labeled with target categories and position information of the corresponding existing targets, and the two target images are registered to eliminate misalignment.

9. A target detection device for cross-domain information fusion, used in the target detection method for cross-domain information fusion according to any one of claims 1 to 8, characterized in that: include: A registration unit, used for registering the modality A image and the modality B image to obtain a registered modality A image and modality B image; A detection unit includes a dual-stream target detection model, which is used to input the registered modality A image and the registered modality B image into the dual-stream target detection model to obtain the detection result. Wherein, the dual-stream target detection model includes: A feature extraction module performs feature extraction on the registered modality A image to obtain a high-level semantic feature A, and performs feature extraction on the registered modality B image to obtain a high-level semantic feature B; A multi-scale enhancement module performs multi-scale enhancement on the feature pair consisting of the high-level semantic feature A and the high-level semantic feature B to obtain an enhanced input feature pair; A fusion module performs spatial-level multimodal interaction and channel-level multimodal interaction on the enhanced input feature pair to obtain multimodal features; The target detection module includes a detection head, which generates a detection frame and a confidence level of a corresponding category as the detection result according to the multimodal features.

Images