Infrared and visible light fusion target detection method and device based on ACF-YOLO

Abstract

The application discloses an infrared and visible light fusion target detection method and device based on ACF-YOLO, which comprises the following steps: acquiring a video stream of a region to be detected; the video stream comprises an infrared video stream and a visible light video stream; inputting the video stream into a target detection model based on ACF-YOLO, sequentially extracting features from vertical, horizontal and global directions through a multi-core three-way perception module of the target detection model, learning the offset of infrared and visible light images through a cross-modal adaptive feature sampling module, sampling and aligning target spatial coordinates, adaptively fusing the features after alignment, and obtaining fusion features; and outputting the position information and the category label of the target according to the fusion features. The target detection method simultaneously solves the problems of modal misplacement and inter-modal interference, and can be widely used in various complex target detection scenes such as automatic driving, intelligent security, industrial manufacturing and aerospace.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and particularly relates to an infrared and visible light fusion target detection method and device based on ACF-YOLO. BACKGROUND

[0002] Target detection has a wide range of applications in industrial inspection, traffic travel and public security. The detection method relying only on visible light imaging is easily affected in reliability and stability in night, backlight, low illumination and shielding environment such as smoke and drizzle; infrared imaging can provide thermal radiation information independent of illumination, but lacks color and fine-grained texture description. Fusion of infrared and visible light is considered an effective way to improve detection performance in complex environments.

[0003] In actual deployment, cross-modal fusion usually faces three key technical obstacles: first, modal misplacement. Due to the differences in imaging mechanism, parallax and distortion, the same target may have a pixel-level position shift between two modalities. If not calibrated effectively before fusion, it is easy to cause positioning drift and missed detection; second, modal interference. When a certain modality is affected by noise or degradation, direct splicing or simple weighting may bring irrelevant information into the fusion representation; third, spatial-semantic mismatch. Deep features weaken spatial details while enhancing semantic abstraction. If there is no corresponding alignment and fusion strategy, it is difficult to balance detection accuracy and stability.

[0004] And the existing technology in the aspect of multi-source fusion generally includes the following routes: one kind of method depends on traditional geometric registration or external parameter calibration and then performs image-level or feature-level superposition; one kind of method adopts early / late fusion and merges multi-modal information before / after backbone extraction; there are methods that introduce adaptive alignment and fusion strategies at the feature level through learning-based alignment and fusion; at the same time, multi-scale prediction is often used to cover targets of different sizes. These methods can achieve certain results under certain conditions, but in complex dynamic scenes (such as motion of airborne platforms, sharp changes in illumination or local occlusion), problems such as insufficient calibration before fusion, sensitivity to noise or insufficient coupling of alignment and prediction may still affect the stability and real-time performance of detection. SUMMARY

[0005] To solve the above technical problems, the present application designs an infrared and visible light fusion target detection device and method based on ACF-YOLO, and deploys it to platforms such as unmanned aerial vehicles, unmanned vehicles and robots, which is suitable for real-time target detection in scenes such as unmanned aerial vehicle inspection, autonomous driving at night and fixed security monitoring.

[0006] To achieve the above object, the application provides an infrared and visible light fusion target detection method and device based on ACF-YOLO.

[0007] A video stream of a region to be detected is acquired, wherein the video stream comprises an infrared video stream and a visible light video stream.

[0008] The video stream is input into a target detection model based on ACF-YOLO, and features are extracted from vertical, horizontal and global directions through a multi-core three-way perception module of the target detection model in sequence, and offsets of infrared and visible light images are learned through a cross-modal adaptive feature sampling module, target spatial coordinates are sampled and aligned, and the aligned features are adaptively fused to obtain fusion features.

[0009] According to the fusion features, position information and class labels of the target are output.

[0010] Preferably, the process of acquiring the video stream of the region to be detected comprises:

[0011] The infrared video stream and the visible light video stream are respectively frame-sampled at a fixed frame rate.

[0012] The resolution of the sampled images is uniformly adjusted to a predetermined size.

[0013] The pixel values are normalized to obtain a standardized image sequence.

[0014] Preferably, the process of inputting the video stream into the target detection model based on ACF-YOLO comprises:

[0015] The standardized image sequence is input into a near-end inference module.

[0016] The near-end inference module is deployed with the target detection model based on ACF-YOLO, and the image sequence is subjected to real-time target detection through the target detection model based on ACF-YOLO, and a detection result containing position information and class labels is output.

[0017] The detection result is structured and packaged through a data regularization module to obtain packaged data containing a timestamp, a video frame identifier, target position and class information.

[0018] The packaged data is transmitted in real time to a remote terminal through a communication link module and is synchronously stored in a result access module.

[0019] Preferably, the process of extracting features from vertical, horizontal and global directions through the multi-core three-way perception module of the target detection model comprises:

[0020] The input feature map is subjected to slicing operation in horizontal, vertical and global directions to obtain sub-feature maps in three directions;

[0021] Partial convolution is performed on the feature maps in three directions to reserve partial channel information for feature extraction, and partial convolution results are obtained;

[0022] The texture features are extracted by the PKIblock unit using multi-scale deep convolution to obtain multi-scale deep convolution results;

[0023] The partial convolution results and the multi-scale deep convolution results are spliced to obtain multi-directional and multi-scale feature representation.

[0024] Preferably, the partial convolution operation comprises:

[0025] The input feature map is convolved by 1 / n, and a 3x3 convolution kernel is used;

[0026] The convolved feature map is spliced to obtain partial convolution feature output.

[0027] Preferably, the process of extracting texture features by the PKIblock unit using multi-scale deep convolution comprises:

[0028] Deep convolution with different convolution kernel sizes is used for n times of feature extraction, and the multi-time deep convolution results are fused to obtain multi-scale texture feature output.

[0029] Preferably, the process of obtaining fused features comprises:

[0030] A two-dimensional spatial offset field between infrared features and visible light features is predicted;

[0031] One of the modal features is subjected to grid resampling according to the two-dimensional spatial offset field to realize pixel-level spatial alignment;

[0032] The aligned features are respectively input into a semantic information branch and a spatial information branch for modeling to obtain semantic enhanced features and spatial consistency features, respectively;

[0033] The semantic enhanced features and the spatial consistency features are adaptively fused to obtain fused features.

[0034] Preferably, the process of inputting the aligned features into the semantic information branch and the spatial information branch for modeling comprises:

[0035] The aligned features are subjected to frequency domain enhancement to obtain frequency domain enhanced features;

[0036] The frequency domain enhanced features and the original semantic features are fused by a gating mechanism to output semantic enhanced features;

[0037] The aligned features are locally geometrically and positionally modeled to obtain spatial consistency features;

[0038] The semantic enhanced features and the spatial consistency features are spliced and subjected to convolution operation to complete adaptive fusion, thereby obtaining fused features.

[0039] Preferably, the ACF-YOLO-based target detection model is deployed on an embedded computing unit of a UAV, an unmanned vehicle or a robot platform.

[0040] The embedded computing unit carries infrared and visible light imaging modules through a three-axis mechanical stabilization holder, so as to reduce the influence of motion jitter on image quality.

[0041] The multi-core three-way perception module is arranged at a shallow stage of a backbone network of the ACF-YOLO-based target detection model, so as to preferentially retain spatial details.

[0042] The cross-modal adaptive feature sampling module is arranged at a neck network of the ACF-YOLO-based target detection model, so as to complete modal alignment and fusion at a deep feature stage.

[0043] The application further provides an infrared and visible light fusion-based target detection device based on ACF-YOLO, comprising:

[0044] A perception input module is configured to acquire a video stream of a region to be detected; the video stream comprises an infrared video stream and a visible light video stream.

[0045] A data regularization module is configured to input the video stream into an ACF-YOLO-based target detection model, extract features from vertical, horizontal and global directions through a multi-core three-way perception module of the target detection model in sequence, learn offsets of infrared and visible light images through a cross-modal adaptive feature sampling module, sample and align target spatial coordinates, and adaptively fuse the aligned features to obtain fused features.

[0046] A proximal inference module is configured to output position information and a class label of a target according to the fused features.

[0047] Compared with the prior art, the application has the following advantages and technical effects:

[0048] The application reconstructs the backbone network by introducing a multi-core three-way perception module, the multi-core three-way perception module extracts multi-scale texture features between different receptive fields by using non-expanded initial deep convolution from different directions, thereby reducing the loss of spatial information in the network, the MAFSM is introduced into the neck network, the network uses a spatial information branch and a semantic information branch to predict the offset of the target, adjusts the spatial coordinates of the object through a sampling operation, and adaptively fuses the features of the infrared image and the visible light image, while solving the problems of modal misplacement and inter-modal interference. The device and method can be widely used in various complex target detection scenes such as automatic driving, intelligent security, industrial manufacturing, aerospace, etc. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. The accompanying drawings should not be construed as an inappropriate limitation on the present application.

[0050] Figure 1 The method flowchart of the embodiment of the present application is shown in the figure;

[0051] Figure 2 The overall framework schematic diagram of the ACF-YOLO of the embodiment of the present application is shown in the figure;

[0052] Figure 3 The module structure schematic diagram of the MTPM of the embodiment of the present application is shown in the figure;

[0053] Figure 4 The insertion position schematic diagram of the MTPM in the backbone network of the embodiment of the present application is shown in the figure;

[0054] Figure 5 The substructure schematic diagram of the PKIBlock and the PConv of the embodiment of the present application is shown in the figure;

[0055] Figure 6 The structure configuration schematic diagram of the MAFSM in the neck network of the embodiment of the present application is shown in the figure;

[0056] Figure 7 The internal structure schematic diagram of the MAFSM of the embodiment of the present application is shown in the figure;

[0057] Figure 8 The device structure schematic diagram of the embodiment of the present application is shown in the figure;

[0058] Figure 9 The data acquisition schematic diagram of the embodiment of the present application applied to a robot is shown in the figure;

[0059] Figure 10 The data acquisition schematic diagram of the embodiment of the present application applied to a self-driving car is shown in the figure;

[0060] Figure 11 Figure 1 is a schematic diagram of data acquisition for a UAV according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0062] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0063] Embodiment one

[0064] As shown in Figure 1 and Figure 2 , the present embodiment provides an infrared and visible light fusion target detection method based on ACF-YOLO, which includes:

[0065] obtaining a video stream of a region to be detected; the video stream includes an infrared video stream and a visible light video stream;

[0066] inputting the video stream into a target detection model based on ACF-YOLO, sequentially extracting features from vertical, horizontal and global directions through a multi-core three-way perception module of the target detection model, reducing spatial information loss and alleviating feature mismatching problems; and learning the offset of infrared and visible light images through a cross-modal adaptive feature sampling module, sampling and aligning the target spatial coordinates, and adaptively fusing the aligned features to obtain fusion features;

[0067] outputting the position information and class label of the target obtained according to the fusion features.

[0068] The present embodiment is based on the ACF-YOLO network, adopts a multi-core three-way perception module (MTPM) to reconstruct the feature extraction backbone module; uses a cross-modal adaptive feature sampling module (MAFSM) to learn the offset of infrared images and visible light images, adjusts the spatial coordinates of the target through a sampling operation, realizes the alignment between the modalities, and adaptively fuses the aligned features based on semantic information and spatial information to solve the interference problem between the modalities.

[0069] The present embodiment uses a multi-core three-way perception module to construct a feature extraction backbone network, and the multi-core three-way perception module (MTPM) uses partial convolution and multi-scale convolution to extract features from vertical, horizontal and global directions, which reduces spatial information loss and alleviates feature mismatching problems.

[0070] The modules of the feature extraction backbone network are in turn:

[0071] Conv, Conv, MTPM, C2f, Conv, C2f, Conv, C2f, Conv, C2f, SPPF.

[0072] Further, the process of acquiring the video stream of the to-be-detected region comprises:

[0073] Frame sampling is performed on the infrared video stream and the visible light video stream respectively at a fixed frame rate;

[0074] The resolution of the sampled images is uniformly adjusted to a predetermined size;

[0075] The pixel values are normalized to obtain a standardized image sequence.

[0076] Further, the embodiment first continuously collects infrared and visible light video streams of the to-be-detected region through platforms such as drones, unmanned vehicles, and robots. After collection, independent frame sampling and image preprocessing are performed on the two video streams. Specifically, for the infrared video stream, frame sampling is performed at a fixed frequency of 30 fps, and the resolution of all sampled images is uniformly adjusted to 640x512. For the visible light video stream, a higher frequency of 60 fps is used for sampling, and then the image resolution is uniformly adjusted to 1080x768, and the pixel value normalization is completed to map all values to the interval [0, 1].

[0077] Further, the process of inputting the video stream into the target detection model based on ACF-YOLO comprises:

[0078] The standardized image sequence is input into the near-end inference module;

[0079] The near-end inference module is deployed with a target detection model based on ACF-YOLO, which performs real-time target detection on the image sequence through the target detection model based on ACF-YOLO, and outputs detection results containing position information and class labels;

[0080] The detection results are structured and packaged through the data regularization module to obtain packaged data containing timestamps, video frame identifiers, target positions, and class information;

[0081] The packaged data is transmitted in real time to the remote terminal through the communication link module and is synchronously stored in the result access module.

[0082] Further, the process of extracting features from vertical, horizontal, and global directions through the multi-core three-way perception module of the target detection model comprises:

[0083] The input feature map is subjected to slicing operations in horizontal, vertical, and global directions to obtain sub-feature maps in the three directions respectively.

[0084] Partial convolution (Pconv) is performed on the feature maps in three directions, preserving some channel information for feature extraction, and the partial convolution result is obtained.

[0085] Texture features are extracted using multi-scale depth convolution through PKIblock units to obtain multi-scale depth convolution results;

[0086] By concatenating some convolution results with multi-scale depth convolution results, multi-directional and multi-scale feature representations are obtained.

[0087] Furthermore, in this embodiment, the feature maps of layers 5, 7, and 10 from the two branches of the feature extraction stage are stitched together. The stitched feature map is then input into the Neck layer for feature extraction. After that, it enters the cross-modal adaptive feature sampling module (MAFSM) to learn the offset between the infrared image and the visible light image. The spatial coordinates of the target are adjusted through sampling operations to achieve alignment between modalities. Based on semantic and spatial information, the aligned features are adaptively fused to solve the problem of interference between modalities.

[0088] Furthermore, some convolution operations include:

[0089] The input feature map is convolved at a ratio of 1 / n, and a 3×3 convolution kernel is used;

[0090] The convolutional feature maps are concatenated to obtain partial convolutional feature output.

[0091] Furthermore, such as Figure 3 As shown, the relevant formulas for some convolutions in this embodiment are as follows:

[0092]

[0093] Where X' represents the features after partial convolution, 1 / n represents the proportion of feature maps using convolution, and in MTPM n=4, Conv 3×3 represents a convolution operation with a kernel of 3, while Concat represents a concatenation operation.

[0094] like Figure 5 As shown, given input features Figure X MTPM uses initial depthwise convolution to extract multi-scale texture features, which can be expressed mathematically as follows:

[0095]

[0096] Where X' represents the feature after depthwise convolution, n represents the number of depthwise convolutions, n=5 in MTPM, and DWConvn represents a depthwise convolution with a kernel size of n.

[0097] Further, the process of extracting texture features by the PKIblock unit using multi-scale deep convolution includes:

[0098] n times of feature extraction are performed using deep convolution with different kernel sizes, and the multi-time deep convolution results are fused to obtain multi-scale texture feature output.

[0099] Further, the PKIblock unit uses non-expanded initial deep convolution to extract multi-scale texture features using different receptive fields.

[0100] Further, as shown in Figure 7 the process of obtaining fused features includes:

[0101] Based on the two-dimensional spatial offset field between the infrared features and the visible light features;

[0102] According to the two-dimensional spatial offset field, one of the modal features is grid resampled to realize pixel-level spatial alignment;

[0103] The aligned features are respectively input into the semantic information branch and the spatial information branch for modeling to respectively obtain semantic enhanced features and spatial consistency features;

[0104] The semantic enhanced features and the spatial consistency features are adaptively fused to obtain fused features.

[0105] Further, the process of respectively inputting the aligned features into the semantic information branch and the spatial information branch for modeling includes:

[0106] The aligned features are frequency domain enhanced to obtain frequency domain enhanced features;

[0107] The frequency domain enhanced features and the original semantic features are fused through a gating mechanism to output semantic enhanced features; the formula is as follows:

[0108]

[0109] The aligned features are locally geometrically and positionally modeled to obtain spatial consistency features;

[0110] The semantic enhanced features and the spatial consistency features are adaptively fused by splicing and convolution operation to obtain fused features.

[0111] The features from the two branches are then connected to generate the final output, and the alignment and adaptive fusion of the RGB image and the IR image are realized. The process is mathematically expressed as:

[0112]

[0113] Wherein, X' represents the final output after MAFSM, and delta represents the predicted offset.

[0114] Further, the ACF-YOLO-based target detection model is deployed on an embedded computing unit of a UAV, an unmanned vehicle or a robot platform.

[0115] The embedded computing unit carries infrared and visible light imaging modules through a three-axis mechanical stabilization holder, for reducing the influence of motion jitter on image quality.

[0116] The multi-core three-way perception module is arranged at a shallow stage of a backbone network of the ACF-YOLO-based target detection model, for preferentially retaining spatial details.

[0117] The cross-modal adaptive feature sampling module is arranged at a neck network of the ACF-YOLO-based target detection model, for completing modal alignment and fusion at a deep feature stage.

[0118] Further, as shown in Figure 6 and Figure 7 , the cross-modal adaptive feature sampling module is specifically arranged after the second downsampling of the neck network, for completing pixel-level alignment of the two modal features before fusion and improving the consistency of cross-scale representation.

[0119] The neck network performs cross-scale upsampling and downsampling connection on the multi-layer features from the backbone network, forming a feature pyramid.

[0120] The detection head performs multi-scale prediction based on the pyramid features, and outputs detection results containing target categories, bounding box coordinates and their confidence.

[0121] Further, the embodiment sends the preprocessed image sequence into the ACF-YOLO network deployed on the UAV, unmanned vehicle, robot and other platforms, performs real-time analysis on the received image data, and quickly generates detection results, which can clearly identify the position information and the belonging category of each target.

[0122] The embodiment first obtains the infrared video stream and the visible light video stream of the to-be-detected region through a perception input module of a UAV, an unmanned vehicle, a robot, etc., performs frame sampling and image preprocessing on the video stream. Then, an infrared and visible light fusion cross-modal target detection network based on ACF-YOLO is constructed. In the network: a multi-core three-way perception module is used to reconstruct a feature extraction backbone network; MAFSM uses a spatial information branch and a semantic information branch to predict the offset of the target, adjusts the spatial coordinates of the object through a sampling operation, and adaptively fuses the features of the infrared image and the visible light image, while solving the problems of modal misplacement and modal interference. Finally, the ACF-YOLO algorithm is deployed and run on a UAV, an unmanned vehicle, a robot, etc. platform, the preprocessed image sequence is taken as input, and the detection result containing the target position information and the category label is output in real time.

[0123] The embodiment proposes a method of “calibration first and fusion later” to solve the three problems of modal misplacement, modal interference and spatial-semantic feature mismatch in infrared and visible light fusion target detection: maintaining multi-scale spatial details before fusion, completing geometric alignment and adaptive fusion of the double channels during the fusion process, to realize robust detection in complex scenes.

[0124] Embodiment two

[0125] As shown in Figure 8 based on the same inventive concept, the embodiment also provides an infrared and visible light fusion target detection device based on ACF-YOLO. The device is mounted on a UAV, an unmanned vehicle, a robot, etc. platform, and each module works cooperatively to complete image processing, target recognition, and output and feedback of the detection result. It includes:

[0126] A perception input module is used to obtain the video stream of the to-be-detected region. The video stream includes an infrared video stream and a visible light video stream.

[0127] A data regularization module is used to input the video stream into the target detection model based on ACF-YOLO. The multi-core three-way perception module of the target detection model extracts features from the vertical, horizontal and global directions in sequence, and the cross-modal adaptive feature sampling module learns the offset of the infrared and visible light images, samples and aligns the target spatial coordinates, adaptively fuses the aligned features, and obtains the fusion features.

[0128] A near-end reasoning module is used to output the position information and the category label of the target based on the fusion features.

[0129] The near-end reasoning module includes an embedded computing unit (K3588), which has the ACF-YOLO algorithm model deployed inside, and is used to perform real-time end-to-end target detection tasks on the standardized image data.

[0130] Further, the target detection device of the embodiment further comprises:

[0131] an airborne support module, an illumination enhancement module, a power supply module, a result access module, and a communication link module;

[0132] The airborne support module is installed at the bottom of the UAV platform, the top of the unmanned vehicle platform, the head of the robot, etc., and is connected with the perception input module and the illumination enhancement module through a three-axis mechanical stabilization gimbal, aiming to reduce the negative impact on the imaging quality during movement.

[0133] The illumination enhancement module is installed on both sides of the three-axis mechanical stabilization gimbal, aiming to provide high-brightness LED fill light in night or low-light environment.

[0134] The power supply module is responsible for providing power support for each power supply module to ensure its stable operation.

[0135] The result access module is specially used for storing algorithm models and corresponding model parameters.

[0136] The function of the result access module is to provide data access support for the entire system, and its storage content includes two categories: one is the algorithm model and related model parameters relied on by the near-end reasoning module; the other is the detection log, structured detection result and original image data generated during the operation of the system.

[0137] The communication link module is responsible for real-time transmission of identification information to the user end.

[0138] The function of the communication link module is to establish a data link between the device and the external terminal. This module is used to transmit the structured identification information encapsulated and arranged by the data normalization module to the remote terminal in real time.

[0139] Further, the airborne support module of the embodiment stably fixes the perception components at the specified position (such as the bottom, top or head) of the UAV, unmanned vehicle, etc. platform through a three-axis mechanical stabilization gimbal, effectively reducing the image jitter caused by movement.

[0140] The perception input module carried by the gimbal is composed of a binocular visible light camera and an infrared thermal imager, providing multi-modal imaging and ranging capability.

[0141] The illumination enhancement module as a matching component is an IPX4 level waterproof high-brightness LED lamp located on both sides of the gimbal, specially providing necessary fill light support for night and low-light environment.

[0142] In addition, the rest of the core components of the device, including the power supply module, data normalization module, near-end reasoning module, result access module and communication link module, are all uniformly arranged in the internal main control cabin of the unmanned aerial vehicle, unmanned vehicle or robot.

[0143] The power supply module is responsible for supplying the necessary electrical energy for each functional component in the device, ensuring stable operation of the entire system.

[0144] The data normalization module is responsible for pre-processing input data and post-processing output results. Specifically, this module is used to receive infrared video streams and visible light video streams from the perception input module and perform pre-processing operations on them, including frame extraction, format conversion, resolution normalization and pixel normalization, thereby generating standardized image data for analysis by the near-end reasoning module. In addition, this module is also used to receive detection results from the near-end reasoning module after the identification task is completed and perform post-processing operations on them, including structured packaging of the detection results, visual labeling and return data arrangement, ultimately forming structured output data.

[0145] Structured packaging is to generate packaging data containing timestamps, corresponding video frame identifiers, target position information and category information, and then transmit it to the result access module for historical record saving.

[0146] Finally, the packaged structured identification information is uniformly sent by the communication link module to ensure that the remote terminal can receive it in real time and accurately.

[0147] As shown in Figure 9 , Figure 10 and Figure 11 , they collectively show the data acquisition application scenarios of the method and device according to the embodiments of the present application on different platforms. In these scenarios, robots ( Figure 9 ), unmanned vehicles ( Figure 10 ) and unmanned aerial vehicles ( Figure 11 ) platforms equipped with the target detection device are respectively shown. These platforms capture scenes containing common targets such as vehicles and pedestrians through the device mounted thereon to synchronously obtain infrared video streams and visible light video streams.

[0148] The embodiment uses a target detection device to obtain an infrared video stream and a visible light video stream of a detection area; inputs the video streams into an ACF-YOLO-based target detection model to obtain a detection result of position information and a category label of a target. The ACF-YOLO-based infrared and visible light fusion cross-modal target detection method uses a multi-core three-way perception module (MTPM) to reconstruct a backbone network, the module uses partial convolution and multi-scale convolution to extract features from vertical, horizontal and global directions, reduces spatial information loss in the feature extraction process, and alleviates the feature mismatch problem, uses a cross-modal adaptive feature sampling module (MAFSM) to learn the offset of infrared images and visible light images, adjusts the spatial coordinates of the target through a sampling operation, realizes alignment between the modes, and adaptively fuses the aligned features on the basis of semantic information and spatial information to solve the interference problem between the modes.

[0149] The device of the embodiment can be flexibly deployed on various platforms such as unmanned aerial vehicles, unmanned vehicles and robots. The system structure is complete, and integrates an on-board support module, an energy supply module, a perception input module, an illumination enhancement module, a data regularization module, a near-end reasoning module, a result access module and a communication link module. The eight core modules work cooperatively to ensure efficient operation of the device in a mobile environment.

[0150] The target detection device based on ACF-YOLO infrared and visible light fusion provided in the embodiment has all the advantages of the target detection method based on ACF-YOLO infrared and visible light fusion provided in the first embodiment.

[0151] Embodiment three

[0152] The embodiment also discloses a computer device, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method in the first embodiment.

[0153] Embodiment four

[0154] The embodiment also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method in the first embodiment.

[0155] Embodiment five

[0156] The embodiment also discloses a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the method in the first embodiment.

[0157] The above merely provides the preferred embodiments of the present application, and the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A target detection method based on ACF-YOLO infrared and visible light fusion, characterized in that, include: Acquire a video stream of the area to be detected; the video stream includes an infrared video stream and a visible light video stream. The video stream is input into the ACF-YOLO-based target detection model. Features are extracted from the vertical, horizontal, and global directions sequentially through the multi-core three-way perception module of the target detection model. The offset between infrared and visible light images is learned through the cross-modal adaptive feature sampling module. The target spatial coordinates are sampled and aligned, and the aligned features are adaptively fused to obtain fused features. The multi-core three-way perception module (MTPM) is set in the backbone network of the ACF-YOLO-based target detection model. A feature extraction backbone network is constructed through the MTPM, and the modules of the feature extraction backbone network are as follows: Conv, Conv, MTPM, C2f, Conv, C2f, Conv, C2f, Conv, C2f, SPPF; The cross-modal adaptive feature sampling module is set after the second downsampling of the neck network of the ACF-YOLO-based target detection model; Based on the fusion features, the target's location information and category label are output; The process of extracting features from the vertical, horizontal, and global directions using the multi-core three-way perception module of the target detection model includes: The input feature map is sliced ​​horizontally, vertically, and globally to obtain sub-feature maps in three directions. Partial convolution is performed on the feature maps in three directions, and feature extraction is performed while retaining some channel information to obtain the partial convolution result; Texture features are extracted using multi-scale depth convolution through PKIblock units to obtain multi-scale depth convolution results; The partial convolution results are concatenated with the multi-scale depth convolution results to obtain multi-directional, multi-scale feature representations; The process of extracting texture features using multi-scale depth convolution through PKIblock units includes: The feature extraction is performed n times using depthwise convolution with different kernel sizes. The results of multiple depthwise convolutions are then fused to obtain multi-scale texture feature output. The process of obtaining fusion features includes: Predict the two-dimensional spatial offset field between infrared and visible light features; Based on the two-dimensional spatial offset field, one modal feature is resampled using a grid to achieve pixel-level spatial alignment. The aligned features are input into the semantic information branch and the spatial information branch respectively for modeling, and semantically enhanced features and spatially consistent features are obtained respectively. The semantic enhancement features and spatial consistency features are adaptively fused to obtain the fused features; The process of inputting the aligned features into the semantic information branch and the spatial information branch for modeling includes: The aligned features are enhanced in the frequency domain to obtain frequency-enhanced features. The frequency domain enhancement features are fused with the original semantic features through a gating mechanism to output semantic enhancement features; Local geometry and positional relationships are modeled on the aligned features to obtain spatially consistent features; By concatenating semantic enhancement features and spatial consistency features and performing convolution operations, adaptive fusion is achieved to obtain fused features.

2. The method according to claim 1, characterized in that, The process of acquiring the video stream of the region to be detected includes: Frame sampling is performed on the infrared video stream and the visible light video stream at a fixed frame rate; The resolution of the sampled images is uniformly adjusted to the predetermined size; Pixel values ​​are normalized to obtain a standardized image sequence.

3. The method according to claim 1, characterized in that, The process of inputting the video stream into the ACF-YOLO-based object detection model includes: The standardized image sequence is input into the near-end inference module; The proximal inference module is equipped with an ACF-YOLO-based target detection model, which performs real-time target detection on image sequences and outputs detection results containing location information and category labels. The detection results are structured and encapsulated using a data normalization module to obtain encapsulated data containing timestamps, video frame identifiers, target locations, and category information. The encapsulated data is transmitted to the remote terminal in real time through the communication link module and simultaneously stored in the result retrieval module.

4. The method according to claim 1, characterized in that, The convolutional operations include: The input feature map is convolved at a ratio of 1 / n, and a 3×3 convolution kernel is used; The convolutional feature maps are concatenated to obtain partial convolutional feature output.

5. The method according to claim 1, characterized in that, The target detection model based on ACF-YOLO is deployed in the embedded computing unit of a drone, unmanned vehicle, or robot platform; The embedded computing unit carries the infrared and visible light imaging modules via a three-axis mechanical stabilization gimbal to reduce the impact of motion jitter on image quality.

6. A target detection device based on ACF-YOLO infrared and visible light fusion, used to implement the method according to any one of claims 1-5, characterized in that, include: The sensing input module is used to acquire the video stream of the area to be detected; The video stream includes an infrared video stream and a visible light video stream; The data normalization module is used to input the video stream into the target detection model based on ACF-YOLO, extract features from the vertical, horizontal and global directions through the multi-core three-way perception module of the target detection model in sequence, learn the offset between infrared and visible light images through the cross-modal adaptive feature sampling module, sample and align the target spatial coordinates, and adaptively fuse the aligned features to obtain fused features. The near-end inference module is used to output the location information and category label of the obtained target based on the fused features.

Images