Target detection method and system based on multi-modal sensor information fusion

Abstract

The invention discloses a multi-modal sensor information fusion-based target detection method and system. The method comprises the steps of obtaining a visible light image and an infrared image of a preset area; based on a depth estimation network, respectively extracting depth information of the visible light image and the infrared image to obtain a depth estimation map after fusion of the visible light image and the infrared image; feature information of the visible light image and the infrared image is extracted based on a feature extraction network, and scene perception features of the depth estimation map are extracted; and carrying out feature fusion on the feature information of the visible light image and the infrared image and the scene perception feature of the depth estimation map by using a feature fusion network to obtain a depth estimation fusion image in a preset region, and carrying out to-be-recognized target detection on the depth estimation fusion image based on a target detection network. Through complementary characteristics of visible light and infrared images, depth information and scene perception capability of the images are enhanced based on a depth estimation network and a feature extraction network, and image target identification accuracy and robustness are improved.

Description

Technical Field

[0001] The present invention relates to the field of security visual detection technology, and in particular to a target detection method and system based on multimodal sensor information fusion. Background Art

[0002] Object detection technology plays an important role in the security field to ensure public safety and protect the safety of important facilities. However, in complex and changing environments, traditional single sensor data may not meet the requirements for accurate detection of targets. In order to make up for the shortcomings of a single sensor, multimodal sensor data fusion has become an effective method. The development of deep learning technology has greatly promoted the progress in the field of object detection; especially in the field of image processing, deep learning algorithms have achieved great success. However, in the field of security, due to factors such as complex scenes and changing lighting conditions, single-modal image data often cannot meet the requirements of accuracy and robustness. Therefore, using multimodal sensor information for object detection has become one of the current research hotspots.

[0003] The use of algorithms to fuse infrared images and visible light images can improve the understanding and perception of the surrounding environment. By combining information from multiple data sources, the model can have more comprehensive and accurate input features, so that high-quality image data can be obtained even in a completely dark environment. Therefore, multimodal data fusion can improve the performance of the model on complex tasks.

[0004] Current target detection methods are mainly based on single sensor data and cannot fully utilize the information of multimodal sensors. The image features collected by a single sensor will vary greatly at different times and in different environments, which poses a challenge to the accuracy and robustness of target detection. Therefore, there is an urgent need for a target detection method that can fully utilize the information of multimodal sensors to improve the accuracy and stability of detection.

[0005] Specifically, current target detection methods usually enhance the visibility of targets by fusing infrared images with visible light images. This fusion method is mainly based on image weighted averaging or feature-level fusion to obtain more informative image data for subsequent processing. Deep learning technology has achieved great success in the field of target detection, especially some classic convolutional neural network (CNN) structures, such as Faster R-CNN and YOLO, which have become the mainstream methods of target detection. The above algorithms mainly achieve target detection by finding the location and category of the target in the image.

[0006] In addition, in the field of deep learning, there are methods that use multimodal data for target detection. These methods usually fuse data collected by different sensors to obtain more comprehensive information. However, current methods often ignore the effective use of depth information, resulting in poor fusion effect. Existing target detection methods often fail to make full use of depth information in the process of multimodal data fusion, which leads to poor fusion effect and affects the accuracy and robustness of target detection. Data differences in different scenarios are not considered. Existing methods also fail to consider the differences in data collected by different sensors in different scenarios and times. This difference may lead to a decrease in fusion effect, which in turn affects the performance of target detection. Summary of the invention

[0007] The purpose of the embodiments of the present invention is to provide a target detection method and system based on multimodal sensor information fusion, which utilizes the complementary characteristics of visible light images and infrared images, enhances the depth information and scene perception ability of the image based on a depth estimation network and a feature extraction network, and improves the recognition accuracy and robustness of the target in the image; by integrating multi-source data, the performance of target detection is optimized, especially under complex or harsh lighting conditions, it can provide more reliable detection results, and enhances the effectiveness and adaptability of the system in practical applications.

[0008] To solve the above technical problems, a first aspect of an embodiment of the present invention provides a target detection method based on multimodal sensor information fusion, comprising the following steps:

[0009] Acquire visible light images and infrared images of a preset area;

[0010] Extracting depth information of the visible light image and the infrared image respectively based on a depth estimation network to obtain a depth estimation image after the visible light image and the infrared image are fused;

[0011] Extracting feature information of the visible light image and the infrared image respectively based on a feature extraction network, and extracting scene perception features of the depth estimation image;

[0012] A feature fusion network is used to fuse the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation image to obtain a depth estimation fusion image in the preset area, and the depth estimation fusion image is used to detect the target to be identified based on the target detection network.

[0013] Furthermore, before respectively extracting the depth information of the visible light image and the infrared image based on the depth estimation network, the method further comprises:

[0014] The visible light image and the infrared image are aligned based on an alignment scale parameter estimation network.

[0015] Furthermore, after performing target detection on the depth estimation fusion image based on the target detection network, the method further includes:

[0016] Obtaining detection result confidence, length and width information, and depth distance information of a plurality of targets to be identified in the depth estimation image;

[0017] Obtaining a first ratio value of the length and width information of the target to be identified corresponding to the maximum value of the confidence level of the detection result to the length and width information of each other target to be identified, and a second ratio value of the depth distance information of the target to be identified corresponding to the maximum value of the confidence level of the detection result to the length and width information of each other target to be identified;

[0018] If the ratio of the first ratio value to the second ratio value corresponding to another target to be identified exceeds a preset distance threshold, it is determined that the detection result of the target to be identified corresponding to the ratio is abnormal.

[0019] Furthermore, the data training set of the object detection network includes depth information of several depth estimation maps.

[0020] Furthermore, the feature extraction network is a convolutional neural network structure, which extracts shallow detail features and deep semantic features of the visible light image and the infrared image respectively through several levels of convolution and pooling operations.

[0021] Furthermore, the depth estimation network extracts depth information of the visible light image and the infrared image through an encoder-decoder structure to obtain a depth estimation map after the visible light image and the infrared image are fused.

[0022] Furthermore, the feature fusion network performs layer-by-layer fusion and weighted summation on the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation image.

[0023] Accordingly, a second aspect of an embodiment of the present invention provides a target detection system based on multimodal sensor information fusion, comprising:

[0024] A multimodal image acquisition module, which is used to acquire visible light images and infrared images of a preset area;

[0025] A depth estimation processing module, which is used to extract the depth information of the visible light image and the infrared image respectively based on a depth estimation network, and obtain a depth estimation map after the visible light image and the infrared image are fused;

[0026] A feature extraction module, which is used to extract feature information of the visible light image and the infrared image respectively based on a feature extraction network, and to extract scene perception features of the depth estimation image;

[0027] The target recognition and detection module is used to use a feature fusion network to fuse the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation image to obtain a depth estimation fusion image in the preset area, and perform target detection on the depth estimation fusion image based on the target detection network.

[0028] Accordingly, a third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned target detection method based on multimodal sensor information fusion.

[0029] Accordingly, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned target detection method based on multimodal sensor information fusion.

[0030] The above technical solution of the embodiment of the present invention has the following beneficial technical effects:

[0031] 1. Make full use of the complementary characteristics of visible light images and infrared images, and enhance the depth information and scene perception capabilities of images through depth estimation networks and feature extraction networks;

[0032] 2. This information is further integrated through the feature fusion network, which improves the recognition accuracy and robustness of the target in the image;

[0033] 3. By integrating multi-source data, the performance of target detection is optimized, especially under complex or harsh lighting conditions, which can provide more reliable detection results and enhance the effectiveness and adaptability of the system in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of a target detection method based on multimodal sensor information fusion provided by an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of a multimodal information fusion detection framework provided by an embodiment of the present invention;

[0036] Figure 3 is a depth estimation network structure diagram provided by an embodiment of the present invention;

[0037] Figure 4 yes Figure 3 The network structure diagram of the ATmodel module;

[0038] Figure 5 It is a module block diagram of a target detection system based on multimodal sensor information fusion provided by an embodiment of the present invention.

[0039] Reference numerals:

[0040] 1. Multimodal image acquisition module, 2. Depth estimation processing module, 3. Feature extraction module, 4. Target recognition and detection module. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0042] Please refer to Figure 1 and Figure 2 A first aspect of an embodiment of the present invention provides a target detection method based on multimodal sensor information fusion, comprising the following steps:

[0043] Step S100, obtaining a visible light image and an infrared image of a preset area.

[0044] Step S200, extracting depth information of the visible light image and the infrared image respectively based on the depth estimation network to obtain a depth estimation image after the visible light image and the infrared image are fused;

[0045] Step S300, extracting feature information of the visible light image and the infrared image respectively based on the feature extraction network, and extracting scene perception features of the depth estimation image;

[0046] Step S400, using a feature fusion network to fuse feature information of the visible light image and the infrared image and scene perception features of the depth estimation image to obtain a depth estimation fusion image in a preset area, and performing target detection on the depth estimation fusion image based on a target detection network.

[0047] Furthermore, before respectively extracting the depth information of the visible light image and the infrared image based on the depth estimation network in step S200, the following steps are included:

[0048] Step S110, aligning the visible light image and the infrared image based on the alignment scale parameter estimation network.

[0049] By aligning the visible light image and the infrared image, the visible light image and the infrared image are kept consistent in subsequent processing. The alignment scale parameter estimation network adopts an alignment algorithm based on a convolutional neural network (CNN) to obtain accurate alignment parameters by training the network to learn the spatial transformation relationship between images.

[0050] The network input is a pair of visible light images and infrared images before alignment. After passing through the convolutional pooling layer and then the fully connected layer, 9 alignment parameters are output. After obtaining the alignment parameters, the perspective transformation in the opencv library is used to align the images.

[0051] This alignment parameter only needs to be run once, and the subsequently acquired visible light images and infrared images will be aligned using this parameter.

[0052] The specific training and implementation details of the above alignment process are as follows:

[0053] Network training:

[0054] Dataset preparation: Pairs of visible and infrared images are collected, which use random parameters for perspective transformation, and artificially obtained unaligned infrared images and alignment parameters.

[0055] Loss function: A regression loss function (MSE mean square error loss) is used to evaluate the difference between the alignment parameters output by the network and the true alignment parameters.

[0056] Optimization algorithm: Adam optimization algorithm is used to optimize network parameters.

[0057] Training process: Step 1: Initialize network parameters. Step 2: Divide the training data into small batches and input them into the network, and calculate the alignment parameters of the network output. Step 3: Calculate the loss function to evaluate the difference between the output alignment parameters and the true alignment parameters. Step 4: Adjust the network parameters through the back-propagation algorithm to minimize the loss function. Step 5: Repeat steps 2-4 until the loss function converges or the preset number of training times is reached.

[0058] Implementation details:

[0059] Convolutional layer and pooling layer: used to extract image features and gradually reduce the spatial dimension.

[0060] Fully connected layer: maps the extracted features to the alignment parameter space.

[0061] Regularization techniques: such as Dropout, to prevent overfitting.

[0062] Please refer to Figure 3 and Figure 4In step S200, the aligned visible light and infrared images are processed using a depth estimation network to obtain a depth estimation map. Compared with the depth estimation network obtained by a single image, this depth estimation map can obtain effective object distance information in different environments such as daytime and nighttime.

[0063] The depth estimation network uses a neural network based on an encoder-decoder structure to extract the depth information of the image through the encoder-decoder structure to achieve the mapping from the image to the depth map. Among them, the input of the depth estimation network is the aligned visible light image and infrared image, with sizes of (H, W, 3) and (H, W, 1) respectively; Encoder: Multi-layer convolutional layers (Conv) and pooling layers (Pooling) to gradually reduce the spatial dimension and extract high-level features; Residual Blocks are used to improve the learning ability and training stability of the network; Skip Connections: Skip connections are established between the encoder and decoder to directly pass the encoder's feature map to the corresponding decoder layer to retain the detail information; Decoder: Multi-layer deconvolution layers (Transposed Conv) and upsampling layers (Upsampling) to gradually restore the spatial dimension and generate a depth map; The final output is a depth estimation map with the same size as the input image. Output layer: Output depth estimation map with a size of (H, W, 1).

[0064] The specific training and implementation details of the depth estimation network are as follows:

[0065] Network training: Collect paired visible light images and infrared images, as well as the corresponding real depth maps, normalize the images and scale the pixel values ​​to the range [0,1]. Randomly crop and flip the images to increase data diversity.

[0066] Loss function: A regression loss function (MSE mean square error loss) is used to calculate the error between the predicted depth map and the true depth map.

[0067] Optimization algorithm: Adam optimization algorithm is used to optimize network parameters.

[0068] Training process: Step 1: Initialize network parameters. Step 2: Divide the training data into small batches and input them into the network, and calculate the depth estimation map output by the network. Step 3: Calculate the loss function and evaluate the error between the output depth map and the true depth map. Step 4: Adjust the network parameters through the back propagation algorithm to minimize the loss function. Step 5: Repeat steps 2-4 until the loss function converges or the preset number of training times is reached.

[0069] Specifically, the feature extraction process in step S300 extracts features of different levels from the visible light image and the infrared image, as well as scene perception features extracted based on the depth estimation map. Features of different levels contain different information. Shallow features are rich in image details, while deep features have stronger semantic information. The feature extraction network adopts the ResNet18 classic convolutional neural network structure, and extracts shallow detail features and deep semantic features through convolution and pooling operations at different levels.

[0070] Specifically, in step S400, the input of different levels of visible light features, infrared features and scene perception features containing depth information are multimodally fused to achieve image data with high resolution and clear texture details in all weather conditions, thereby improving the subsequent feature extraction effect and enhancing the model detection capability. The feature fusion network adopts the CNN convolutional neural network structure to perform layer-by-layer fusion and weighted summation of features of different modalities to ensure full utilization and complementarity of information.

[0071] Furthermore, after performing target detection on the depth estimation fusion image based on the target detection network in step S400, the method further includes:

[0072] Step S410, obtaining detection result confidence, length and width information, and depth distance information of a plurality of targets to be identified in the depth estimation image.

[0073] Step S420, obtaining a first ratio value of the length and width information of the target to be identified corresponding to the maximum value of the confidence level of the detection result and the length and width information of each other target to be identified, and a second ratio value of the depth distance information of the target to be identified corresponding to the maximum value of the confidence level of the detection result and the length and width information of each other target to be identified;

[0074] Step S430: If the ratio of the first ratio value to the second ratio value corresponding to another target to be identified exceeds a preset distance threshold, it is determined that the detection result of the target to be identified corresponding to the ratio is abnormal.

[0075] The depth map obtained by depth estimation is used to constrain the target size obtained by the target detection network. The target distance is inversely proportional to the size, which effectively reduces the false alarm targets whose target distance and scale do not match, and improves the detection capability of the detection network.

[0076] Furthermore, the data training set of the target detection network includes the depth information of several depth estimation maps. When training the target detection network, the depth information can be added as an auxiliary constraint to improve the accuracy and robustness of the detection results. The target detection network can use the YOLOv10 target detection algorithm to achieve accurate detection of the target by performing detection box regression and classification on the fused features.

[0077] Furthermore, the feature extraction network is a convolutional neural network structure, which extracts shallow detail features and deep semantic features of visible light images and infrared images respectively through several levels of convolution and pooling operations.

[0078] The feature extraction network adopts a convolutional neural network structure, and uses several levels of convolution and pooling operations to process visible light images and infrared images respectively, which has many advantages. On the one hand, it can effectively extract shallow detail features of the image, retain rich image texture, edge and other information, and provide fine basic data for subsequent analysis and processing. On the other hand, it can also extract deep semantic features, deeply understand the essential meaning and content of the image, and help achieve more advanced image recognition, classification and other tasks. This method fully combines the advantages of shallow details and deep semantics, and provides a powerful and comprehensive feature representation for the processing of visible light images and infrared images, thereby improving the accuracy and reliability of image processing.

[0079] Furthermore, the depth estimation network extracts the depth information of the visible light image and the infrared image through an encoder-decoder structure to obtain a depth estimation map after the visible light image and the infrared image are fused.

[0080] The depth estimation network uses an encoder-decoder structure to extract depth information from visible light images and infrared images, and then obtain a fused depth estimation map. This structure can fully exploit the depth clues in visible light images and infrared images. The encoder can efficiently compress image information and extract key features, while the decoder can gradually restore the details and depth information of the image. The depth estimation of the fusion of visible light and infrared images can comprehensively utilize the advantages of both images. Visible light images usually have rich texture and color information and can provide detailed scene descriptions. Infrared images are sensitive to temperature differences and have unique advantages in low light or complex environments. The depth estimation map obtained in this way is more accurate and reliable, which improves the overall performance and adaptability of the image processing system.

[0081] Furthermore, the feature fusion network performs layer-by-layer fusion and weighted summation of the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation map. The above can fully integrate information from different sources, and closely combine the rich color and texture features of the visible light image, the unique thermal radiation features of the infrared image, and the scene perception features of the depth estimation map. Layer-by-layer fusion allows the features of different levels to be gradually integrated, ensuring the meticulousness and comprehensiveness of the fusion process. The weighted summation can be flexibly adjusted according to the importance of different features, highlighting key information and suppressing irrelevant or noise information. Ultimately, the obtained fused features have richer expressiveness and stronger adaptability, providing a high-quality feature foundation for subsequent image analysis and processing tasks, and helping to improve the accuracy and robustness of tasks such as image recognition and target detection.

[0082] Layer-by-layer fusion refers to the process of gradually integrating various features according to different levels in the network structure during feature fusion. Specifically:

[0083] Feature extraction stage:

[0084] First, the feature extraction network (such as convolutional neural network) extracts feature information from visible light images, infrared images, and depth estimation images. Since different image modalities have different characteristics, the feature extraction network is often designed as a deep structure that can extract features at different levels layer by layer:

[0085] Shallow features: usually contain details (such as edges, textures, etc.) and reflect the local characteristics of the image.

[0086] Deep features: capture higher-level semantic information, which can help the model better understand the overall information such as objects and scenes in the image.

[0087] Weighted summation is a form of assigning different weights to different features during the feature fusion process, thereby achieving comprehensive evaluation and integration of features, as follows:

[0088] Weight assignment: When performing feature fusion, different weights are assigned to each type of feature to reflect its importance in the feature fusion process. The weight setting can be obtained through static preset, training learning, or dynamic adjustment. For example, in complex scenes, the depth estimation map may provide more important location information, so its weight can be set higher; relatively speaking, if a certain feature is less important in the current task, its weight can be lowered.

[0089] Sum operation: Multiply the weight of each modal feature by the corresponding feature information to form a weighted feature representation. Finally, add these weighted features to obtain a fused feature vector.

[0090] Accordingly, please refer to Figure 5 A second aspect of an embodiment of the present invention provides a target detection system based on multimodal sensor information fusion, comprising:

[0091] A multimodal image acquisition module 1, which is used to acquire a visible light image and an infrared image of a preset area;

[0092] A depth estimation processing module 2 is used to extract the depth information of the visible light image and the infrared image respectively based on the depth estimation network, and obtain a depth estimation map after the visible light image and the infrared image are fused;

[0093] A feature extraction module 3 is used to extract feature information of the visible light image and the infrared image respectively based on a feature extraction network, and to extract scene perception features of the depth estimation image;

[0094] The target recognition and detection module 4 is used to use the feature fusion network to fuse the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation image to obtain a depth estimation fusion image in a preset area, and perform target detection on the depth estimation fusion image based on the target detection network.

[0095] The above-mentioned target detection system can be further expanded into various functional units according to the refinement of the corresponding method steps to execute the various steps corresponding to the refined method.

[0096] Accordingly, a third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned target detection method based on multimodal sensor information fusion.

[0097] Accordingly, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned target detection method based on multimodal sensor information fusion.

[0098] The embodiment of the present invention aims to protect a method and system for target detection based on multimodal sensor information fusion, wherein the method includes the following steps: obtaining a visible light image and an infrared image of a preset area; extracting the depth information of the visible light image and the infrared image respectively based on a depth estimation network to obtain a depth estimation map after the visible light image and the infrared image are fused; extracting the feature information of the visible light image and the infrared image respectively based on a feature extraction network, and extracting the scene perception features of the depth estimation map; using a feature fusion network to fuse the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation map to obtain a depth estimation fusion image in the preset area, and detecting the target to be identified on the depth estimation fusion image based on the target detection network. The above technical solution has the following effects:

[0099] 1. Make full use of the complementary characteristics of visible light images and infrared images, and enhance the depth information and scene perception capabilities of images through depth estimation networks and feature extraction networks;

[0100] 2. This information is further integrated through the feature fusion network, which improves the recognition accuracy and robustness of the target in the image;

[0101] 3. By integrating multi-source data, the performance of target detection is optimized, especially under complex or harsh lighting conditions, which can provide more reliable detection results and enhance the effectiveness and adaptability of the system in practical applications.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0103] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A target detection method based on multimodal sensor information fusion, characterized in that: The steps include: Acquire visible light images and infrared images of a preset area; Extracting depth information of the visible light image and the infrared image respectively based on a depth estimation network to obtain a depth estimation image after the visible light image and the infrared image are fused; Extracting feature information of the visible light image and the infrared image respectively based on a feature extraction network, and extracting scene perception features of the depth estimation image; A feature fusion network is used to fuse the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation image to obtain a depth estimation fusion image in the preset area, and the depth estimation fusion image is used to detect the target to be identified based on the target detection network.

2. The target detection method based on multimodal sensor information fusion according to claim 1, characterized in that: Before respectively extracting the depth information of the visible light image and the infrared image based on the depth estimation network, the method includes: The visible light image and the infrared image are aligned based on an alignment scale parameter estimation network.

3. The target detection method based on multimodal sensor information fusion according to claim 1, characterized in that: After the target detection network is used to detect the target to be identified on the depth estimation fusion image, the method further includes: Obtaining detection result confidence, length and width information, and depth distance information of a plurality of targets to be identified in the depth estimation image; Obtaining a first ratio value of the length and width information of the target to be identified corresponding to the maximum value of the confidence level of the detection result to the length and width information of each other target to be identified, and a second ratio value of the depth distance information of the target to be identified corresponding to the maximum value of the confidence level of the detection result to the length and width information of each other target to be identified; If the ratio of the first ratio value to the second ratio value corresponding to another target to be identified exceeds a preset distance threshold, it is determined that the detection result of the target to be identified corresponding to the ratio is abnormal.

4. The target detection method based on multimodal sensor information fusion according to claim 3 is characterized in that: The data training set of the object detection network includes depth information of a plurality of the depth estimation maps.

5. The target detection method based on multimodal sensor information fusion according to claim 1, characterized in that: The feature extraction network is a convolutional neural network structure, which extracts shallow detail features and deep semantic features of the visible light image and the infrared image respectively through several levels of convolution and pooling operations.

6. The target detection method based on multimodal sensor information fusion according to claim 1, characterized in that: The depth estimation network extracts the depth information of the visible light image and the infrared image through an encoder-decoder structure to obtain a depth estimation map after the visible light image and the infrared image are fused.

7. The target detection method based on multimodal sensor information fusion according to claim 1, characterized in that: The feature fusion network performs layer-by-layer fusion and weighted summation on the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation map.

8. A target detection system based on multimodal sensor information fusion, characterized in that: include: A multimodal image acquisition module, which is used to acquire visible light images and infrared images of a preset area; A depth estimation processing module, which is used to extract the depth information of the visible light image and the infrared image respectively based on a depth estimation network, and obtain a depth estimation map after the visible light image and the infrared image are fused; A feature extraction module, which is used to extract feature information of the visible light image and the infrared image respectively based on a feature extraction network, and to extract scene perception features of the depth estimation image; The target recognition and detection module is used to use a feature fusion network to fuse the feature information of the visible light image and the infrared image and the scene perception features of the depth estimation image to obtain a depth estimation fusion image in the preset area, and perform target detection on the depth estimation fusion image based on the target detection network.

9. An electronic device, characterized in that: include: at least one processor; And a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the target detection method based on multimodal sensor information fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, a target detection method based on multimodal sensor information fusion as described in any one of claims 1-7 is implemented.

Images