In-vehicle target detection method and system based on binocular multi-modal fusion and two-stage attention mechanism

Abstract

The invention belongs to the technical field of intelligent driving assistance systems, and particularly relates to an in-vehicle target detection method and system based on binocular multi-modal fusion and a two-stage attention mechanism. According to the method, RGB and IR video streams are synchronously collected through a binocular camera, and invalid frames are removed through time synchronization buffering and quality evaluation; in the first stage, an attention thermodynamic diagram is generated by using lightweight ResNet, ROI is extracted, and target features are weighted and enhanced; carrying out geometric registration on RGB and IR images, and adaptively selecting a parallel multi-modal fusion path according to an illumination index; in the second stage, an improved YOLO network is combined with cross-scale feature fusion, Anchor and dynamic reasoning scheduling detection targets are optimized, and a detection result is subjected to weighted fusion and time consistency optimization. According to the method, the problems of missing detection of small targets in the vehicle, poor environmental adaptability and high calculation delay are effectively solved, and high-precision and high-robustness real-time detection is realized on vehicle gauge level hardware.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving assistance system technology, and in particular relates to an in-vehicle target detection method and system based on binocular multimodal fusion and a two-stage attention mechanism. Background Technology

[0002] With the rapid development of artificial intelligence, computer vision, and deep learning technologies, vehicle behavior detection has become one of the important research directions in the fields of intelligent transportation systems and autonomous driving. Vehicle behavior detection can not only effectively improve traffic safety and prevent accidents, but also provide drivers with intelligent driver assistance functions and optimize the driving experience. In recent years, deep learning, especially convolutional neural networks, has made significant progress in image recognition and object detection. In particular, object detection methods based on the YOLO series of models have been widely used in real-time detection tasks due to their high detection performance and low computational cost.

[0003] In existing technologies, the following methods are typically used for detecting targets inside a vehicle:

[0004] (1) Detection methods based on rules and classical vision

[0005] Rule-based or traditional image processing methods can effectively perform vehicle behavior detection or driver state recognition. These methods do not require large-scale system training; instead, they rely on manually designed features and threshold rules for judgment. For example:

[0006] Color threshold segmentation: By using skin color or a specific color range of an object, the video image is binarized to segment the driver's hands, face, or specific objects (such as cigarettes, mobile phones, etc.).

[0007] Edge detection and contour tracking: Image edges are extracted using operators such as Canny and Sobel, and contour tracking algorithms are combined to identify the position of the seat belt or the trajectory of hand movements;

[0008] For example, driving behavior can be determined by using fixed sensors to collect head posture, eye movement features, or hand trajectories.

[0009] (2) Multimodal fusion method

[0010] Multimodal fusion technology refers to the comprehensive processing of sensor information from different sources and types to obtain more accurate and robust driver behavior monitoring results than single-modal analysis. In the field of in-vehicle driver monitoring, relying on the visual modality (RGB video stream) collected by in-vehicle cameras, the system can detect the driver's face, eyes, hands, body posture, and interaction with objects (such as mobile phones and cigarettes). At the same time, the audio modality is collected by in-vehicle microphones to analyze the driver's speaking behavior, tone changes, emotional tendencies, and even distracted conversations. In addition, in-vehicle sensors (such as seat belt buckle sensors, seat pressure sensors, steering wheel torque sensors, and acceleration sensors) can provide key information such as occupant position, seat belt status, and operating force in real time, while environmental sensors (such as in-vehicle and out-of-vehicle temperature and humidity meters and light sensors) can help improve the reliability of the recognition results under different external conditions.

[0011] Regarding multi-source data fusion strategies, decision-level fusion is adopted. This means that each modality independently completes the detection and classification tasks, and the resulting judgments are then synthesized based on a voting mechanism, weighted average, or confidence threshold. Figure 1 As shown, taking the detection of whether a driver is smoking as an example, the system can identify the driver's hand and object shape features in the visual modality to determine whether they are holding a cigarette. Simultaneously, it performs parallel detection using in-vehicle aerosol or particulate matter concentration sensors. When visual detection is obstructed or there is insufficient lighting, the sensor modality can still provide effective judgment, thus ensuring that the system retains strong recognition capabilities even when any modality fails. This multimodal fusion method not only significantly enhances the robustness and fault tolerance of vehicle behavior detection in complex lighting, obstruction, and variable driving environments, but also allows different information sources to complement each other, improving overall detection accuracy and adaptability.

[0012] (3) Deep learning model recognition method

[0013] In recent years, with the rapid development of deep learning technology, object detection methods based on Convolutional Neural Networks (CNNs) have been widely applied in the fields of vehicle behavior detection and driver state recognition, such as... Figure 2As shown. These methods rely on deep neural network structures, enabling automatic feature extraction and classification in an end-to-end manner. This eliminates the reliance on manually designed features found in traditional methods, effectively improving the system's generalization ability and recognition accuracy in complex environments. A typical example is the two-stage detection method Faster R-CNN, which uses a Region Proposal Network (RPN) to quickly generate candidate regions, and then uses classification and regression networks to accurately locate and identify targets. This method exhibits high accuracy in small target detection and complex backgrounds; for example, it can accurately identify cigarettes in a driver's mouth, mobile phones in their hands, or facial features under partially occluded conditions in in-vehicle scenarios. These model-based recognition methods offer significant accuracy improvements compared to traditional methods and have become the mainstream technology for in-vehicle behavior detection.

[0014] The existing methods described above have the following shortcomings:

[0015] (1) Rule-based and classical vision-based detection methods rely on manually set features and threshold rules, which are poorly adaptable to situations such as differences in driver skin color, changes in object appearance, changes in lighting intensity, and partial occlusion. During vehicle operation, when lighting conditions change rapidly (such as entering or exiting tunnels) or when there are complex backgrounds or when passengers' clothing is similar in color to the target, it is easy to cause false detection or missed detection of the target area. In addition, these methods generally lack adaptive capabilities and cannot dynamically adjust parameters according to changes in the environment and individual drivers, thus affecting their robustness and generalization in real road environments.

[0016] (2) Although multimodal fusion can maintain a certain recognition capability when a single modality fails, most existing fusion strategies are decision-level fusion strategies, that is, each modality is processed independently and then the results are fused. In this way, when a detection error occurs within each modality, the error information will directly affect the final decision, and there is a lack of deeper feature sharing and collaborative optimization. In addition, multimodal systems require the installation of multiple types of sensors in actual vehicle deployment, as well as time synchronization and data alignment, resulting in high hardware and system maintenance costs. At the same time, the reliability of the fusion results will decrease significantly when sensor data is inaccurate or delayed.

[0017] (3) While current deep learning-based detection methods outperform traditional methods in terms of accuracy, their training process relies on a large number of accurately labeled and diverse datasets, resulting in high data acquisition and labeling costs. Furthermore, their deployment is highly dependent on computing hardware (e.g., requiring accelerators such as GPUs and NPUs), increasing system cost and power consumption. In real-time applications, especially when processing high-definition video streams or multi-camera inputs, model inference latency may not meet the low-latency security requirements. In addition, when training data differs from actual vehicle scenarios (e.g., different vehicle layouts or camera installation locations), the model's generalization ability may still be affected, requiring additional transfer learning or fine-tuning.

[0018] Therefore, there is an urgent need to design a detection method that is suitable for small-sized targets in vehicle-mounted scenarios and has high detection accuracy and significant performance. Summary of the Invention

[0019] The technical problem to be solved by this invention is to design an in-vehicle target detection method and system based on binocular multimodal fusion and two-stage attention mechanism. It simultaneously achieves high-precision detection of small targets, high robustness and low latency real-time operation on an automotive-grade embedded platform, effectively solving the problems of high false negative rate of small targets, poor environmental adaptability and high computational cost in existing vehicle detection technologies.

[0020] To address the aforementioned technical problems, this invention discloses an in-vehicle target detection method based on binocular multimodal fusion and a two-stage attention mechanism, specifically including the following steps:

[0021] Step S1: Acquire RGB video streams and IR video streams through a visible light camera and an infrared camera. The exposure time and frame rate of the two cameras are synchronized, and a unified timestamp mechanism is used at the acquisition end to match and align the acquired RGB and IR frames one by one.

[0022] Step S2: Perform a first-stage coarse detection on the RGB video frames to generate an attention heatmap, extract the region of interest (ROI), and perform feature enhancement and background suppression on the ROI to obtain a weighted RGB image.

[0023] Step S3: Perform geometric registration on the weighted RGB image and the synchronized IR image, and perform multimodal fusion on the weighted RGB image and the registered IR image. Select the fusion path according to the ambient light index to form a multi-channel input.

[0024] Step S4: Input the fused image into the improved YOLO detection network for the second stage of fine detection, and then weight and fuse the output confidence of the YOLO detection network with the statistical scores of the heatmap region in the first stage to obtain the final detection result.

[0025] In a further step of this invention, in step S1, a ring buffer queue is implemented at the acquisition end to buffer RGB and IR video frames respectively. The frame alignment threshold is dynamically adjusted by a timestamp matching algorithm to ensure that the output frame pairs can be transmitted to the downstream processing module within a synchronization error range of ±1 / 2 frame period.

[0026] Furthermore, in step S1 of this invention, an image quality assessment module is added to the acquisition thread to perform motion blur detection (based on gradient variance) and exposure anomaly detection (based on histogram distribution) on the frames, automatically removing obviously blurred or overexposed / underexposed frames, reducing the amount of invalid data in subsequent processing, and improving the overall detection accuracy and real-time performance.

[0027] Furthermore, in this invention, step S2 specifically includes the following steps:

[0028] Step S21: Synchronize the RGB video frames Input to a trained lightweight ResNet network or an equivalent convolutional neural network Extract its multi-channel intermediate feature map. , , Then, the attention heatmap is obtained by dimensionality reduction through 1×1 convolution and concatenation with the Sigmoid activation map:

[0029]

[0030] in, This indicates the probability that the corresponding pixel belongs to the target object (such as a mobile phone, cigarette, or seat belt). For activation function, For a 1×1 convolution kernel in the th... The weight parameters corresponding to each channel For the input feature map at the th Each channel, location eigenvalues ​​at that location For bias terms, The range of values ​​is .

[0031] Step S22: After obtaining the heat map, set a threshold. Binary segmentation is performed on the thermal values. The range of values ​​is Obtain the mask of the potential target area. :

[0032]

[0033] Step S23: Extract candidate region set through connected component analysis Each candidate region is defined as a Region of Interest (ROI) in the form of a minimum bounding rectangle. To enhance the features of the candidate regions and suppress background interference in subsequent detection, the set of ROIs is mapped to an attention weight mask. :

[0034]

[0035] in, The enhancement factor for the ROI region. The suppression coefficient for the background region;

[0036] Step S24: Apply the weight mask Applying the weighted method point by point to the original RGB image yields the weighted image. :

[0037]

[0038] Furthermore, in step S3 of this invention, the registered RGB image With IR plot Multimodal fusion can be performed using the following two strategies:

[0039] Path A fusion (RGB-dominant): The attention-weighted RGB three-channel image and the registered IR single-channel image are directly stacked along the channel dimension. The resulting H×W×4 multi-channel input is fed into the YOLO backbone network.

[0040] Path B fusion (IR-dominant): Expand the IR image into a three-channel base map, and then fuse it with the RGB single-channel luminance map in the channel dimension. Stacking creates another four-channel input:

[0041] Real-time calculation of the current ambient light index :

[0042]

[0043] in, This is the average RGB brightness (usually derived from camera sensor data, typically ranging from 0 to 255; normalization to 0 to 1 is recommended). Infrared contrast (IR Contrast, which reflects the effectiveness of infrared supplementary lighting, is usually obtained through infrared sensors or image analysis, and it is recommended to normalize the range to 0~1). The average RGB brightness weight reflects the "sufficiency of visible light". Infrared contrast weights reflect "infrared confidence level". .

[0044] when Choose path A at this time;

[0045] when Choose path B;

[0046] when At that time, path A and path B are computed in parallel;

[0047] in, This is the high light threshold; anything above it is considered "sufficient visible light". The ambient light index is the low light threshold; anything below it is considered "insufficient visible light." The closer the index is to 1, the more sufficient the ambient light is, and infrared supplementary lighting is not required; Ambient Light Index The closer it is to 0, the weaker the ambient light is, and infrared supplemental lighting is required.

[0048] Furthermore, in step S4 of this invention, the detection network employs an improved YOLO series single-stage detector:

[0049] At the model structure level, the input convolutional kernels of the YOLO backbone network were adjusted to adapt... Channel input:

[0050]

[0051] Simultaneously, a cross-scale feature fusion path is introduced into the Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) structures:

[0052]

[0053] in, Represents the set of adjacent scales. For upsampling / downsampling alignment operation, These are learnable weights used to enhance the multi-resolution representation of features of small targets (such as cigarettes and mobile phones).

[0054] Furthermore, in step S4 of this invention, the K-means++ clustering algorithm is used to optimize the anchor size to match the scale of small targets inside the vehicle. Specifically, the K-means++ clustering algorithm is used to optimize the anchor size in the training set of labeled boxes. Minimize the average Distance, to obtain the optimal set of Anchor sizes The optimization objective is:

[0055]

[0056] Among them, intersection and union ratio The calculation formula is: This optimization ensures that the Anchor is highly matched to the scale of the real target inside the vehicle, thereby significantly reducing the bounding box localization error.

[0057] Furthermore, in step S4 of this invention, during the detection result fusion stage, the software layer will output the confidence level of YOLO. Compared with the first stage heatmap score Double weighting:

[0058]

[0059] in, To determine the final confidence score, The target confidence score output by the YOLO master detection network. This represents the statistical score of the first-stage attention heatmap within the detection bounding box region. For dynamic adjustment coefficients, It can be dynamically adjusted according to factors such as ambient light and occlusion rate to ensure the robustness of the test results. The range of values ​​is The final detection results will also enter a cache pool for time consistency optimization, eliminating false detections in single frames and compensating for short-term missed detections.

[0060] Furthermore, in this invention, the training method for the attention network used in the first stage of coarse detection in step S2 includes:

[0061] Step 1: Build a training dataset and generate GT (Ground Truth) heatmap labels using professional annotation tools.

[0062] Step 2: Label the target region ROI for each frame in the training dataset and convert the ROI into a probabilistic two-dimensional Gaussian distribution heatmap to guide the attention network to learn the spatial distribution of possible target regions.

[0063] Step 3: Use the Binary Cross Entropy (BCE) loss to calculate the predicted heatmap. Compared with the actual heat map Differences between them:

[0064] ;

[0065] Where N is the total number of pixels in the heatmap. is the BCE loss value, which can maintain high sensitivity even when the ratio of positive to negative samples is unbalanced, which is beneficial to improving high recall. i is the pixel index.

[0066] In a further step of this invention, the training dataset includes publicly available driver behavior detection datasets (such as Drive&Act, AUC Distracted Driver, SafeDrive, etc.) and self-collected in-vehicle multi-environment video frames. The self-collected video frames cover strong light, weak light, backlight conditions and behavior categories such as making phone calls, smoking, looking down, and drinking water.

[0067] Furthermore, in step S4 of this invention, the training and optimization strategy for the YOLO main detection network in the second stage includes:

[0068] A multimodal joint training strategy is adopted, which allows RGB and IR to be input into the detection network simultaneously in the same batch. The information of both modalities is retained throughout the entire process from feature extraction to detection prediction, thereby improving the robustness of the network under low light, strong light, complex background and local occlusion conditions.

[0069] To address the high risk of missed detection for small objects inside vehicles (such as cigarettes and mobile phones), Focla Loss is introduced in the classification branch to give higher weights to samples that are more difficult to detect.

[0070]

[0071] in, The Focla Loss value is... This is a class balancing factor used to assign different weights to positive and negative samples in an imbalanced dataset. To focus parameters, The difficulty level of the sample is adjusted by a factor that, when close to 0, reduces the sample's contribution to the loss, while a factor that, when close to 1, retains a high loss weight.

[0072] In the high-resolution feature layer where small targets frequently appear, denser small-sized anchors are added, and a spatial attention mechanism is incorporated into this feature layer to enhance the feature representation of small targets.

[0073] The optimization process employs a training strategy based on SGD, combined with cosine annealing learning rate scheduling and a warm-up phase, enabling the network to converge smoothly in the early stages and making the gradients more adaptive in the later stages.

[0074] Simultaneously, mixed precision training (FP16) is used to improve computational efficiency and memory utilization, thereby ensuring a significant improvement in detection accuracy, especially for small targets, while maintaining the inference speed within the range that automotive-grade embedded platforms can run in real time.

[0075] This invention also provides an in-vehicle target detection system based on binocular multimodal fusion and a two-stage attention mechanism, employing the aforementioned in-vehicle target detection method based on binocular multimodal fusion and a two-stage attention mechanism, specifically including the following modules:

[0076] The binocular acquisition module includes a visible light camera and an infrared camera, used to simultaneously acquire RGB video streams and IR video streams inside the vehicle.

[0077] The synchronization optimization module includes a circular buffer unit and a quality evaluation unit. The circular buffer unit is used to cache RGB frames and IR frames and achieve timing synchronization through timestamp matching. The quality evaluation unit is used to detect motion blur and exposure abnormalities in video frames and remove invalid frames.

[0078] The attention generation module is used to input synchronized RGB video frames into a lightweight ResNet network, generate attention heatmaps and attention weight masks, and perform weighted processing on the original RGB images.

[0079] The multimodal fusion module is used to perform geometric registration on weighted RGB images and IR images, calculate the ambient illumination index, select or execute the corresponding feature fusion path in parallel, and output multi-channel fused data.

[0080] The target detection module includes an improved YOLO detection network, an inference scheduling unit, and a fusion optimization unit. The improved YOLO detection network is used to perform target detection on multi-channel fused data. The inference scheduling unit is used to dynamically switch detection modes. The fusion optimization unit is used to perform weighted fusion and time consistency optimization on the detection results and output the final detection results.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] (1) This invention proposes an in-vehicle driver behavior and object recognition method based on binocular multimodal two-stage attention detection. Through hardware acquisition synchronous optimization + one-stage attention ROI focusing + two-stage cross-scale high-precision detection + illumination adaptive multimodal fusion + intelligent inference scheduling + targeted training optimization, it simultaneously achieves high-precision detection of small targets, high robustness and low latency real-time operation on automotive-grade embedded platforms. It effectively solves the problems of high small target false negative rate, poor environmental adaptability and high computational cost in existing vehicle detection technologies.

[0083] (2) The present invention reduces timing mismatch error by using a binocular (RGB+IR) hardware synchronous acquisition and timestamp matching mechanism (±1 / 2 frame period accuracy).

[0084] (3) The present invention introduces a real-time quality assessment and frame rejection strategy, which effectively filters motion blur and overexposed / underexposed frames, improves the clarity and effective information of the input image at the source, and enables subsequent detection to run on stable, high-quality data.

[0085] (4) The dual-path fusion strategy (RGB main path A, IR main path B) adopted in this invention can be adaptively selected or run in parallel according to the ambient light conditions, ensuring stable detection even in extreme environments such as strong light, weak light, and backlight. Attached Figure Description

[0086] The specific embodiments of the present invention will be further explained below with reference to the accompanying drawings.

[0087] Figure 1 This is a schematic diagram of decision-level fusion used in existing multimodal fusion methods.

[0088] Figure 2 This is a flowchart illustrating the application of existing convolutional neural network-based target detection methods in vehicle behavior detection and driver state recognition.

[0089] Figure 3 This is a flowchart of the in-vehicle target detection method based on binocular multimodal fusion and a two-stage attention mechanism in this invention.

[0090] Figure 4 This is a schematic diagram of binocular video stream acquisition and software optimization in Embodiment 1 of the present invention.

[0091] Figure 5 This is a schematic diagram of the first stage of coarse detection and attention region generation in step S2 of the present invention.

[0092] Figure 6 This is a flowchart of the multi-module feature fusion in Embodiment 1 of the present invention.

[0093] Figure 7 This is a schematic diagram of the YOLO model for the second stage of main detection in Embodiment 1 of the present invention.

[0094] Figure 8 This is the overall flowchart of Embodiment 1 of the present invention.

[0095] Figure 9 This is a system block diagram of the in-vehicle target detection system based on binocular multimodal fusion and two-stage attention mechanism of the present invention. Detailed Implementation

[0096] Example 1

[0097] Combination Figure 3 As shown, the in-vehicle target detection method based on binocular multimodal fusion and a two-stage attention mechanism in this embodiment specifically includes the following steps:

[0098] Step S1: Acquire RGB and IR video streams using a visible light camera and an infrared camera. The exposure time and frame rate of both cameras are synchronized, and a unified timestamp mechanism is used at the acquisition end to match and align the acquired RGB and IR frames one-to-one. The RGB video stream is used to capture the driver's appearance, object colors, and detailed textures under natural light or artificial lighting conditions inside the vehicle. The IR video stream is used to provide the outline shape and edge structure information of the human body and objects in low-light environments (such as at night) or strong backlight environments, reducing the impact of changes in visible light conditions.

[0099] Specifically, this embodiment introduces several optimization mechanisms in the video acquisition stage: (1) time synchronization buffer mechanism; (2) real-time quality assessment and frame discarding mechanism.

[0100] In this embodiment, preferably, in step S1, a ring buffer queue is implemented at the acquisition end to buffer RGB and IR video frames respectively. The frame alignment threshold is dynamically adjusted through a timestamp matching algorithm to ensure that the output frame pairs can be transmitted to the downstream processing module within a synchronization error range of ±1 / 2 frame period.

[0101]

[0102] in, and This is the hardware timestamp of the current frame of the RGB / IR stream, recorded by the acquisition driver at the moment the frame is interrupted, typically based on the system startup clock or PTP synchronization clock. This is the dynamic frame alignment threshold. Specifically, in this embodiment, .

[0103] This mechanism can automatically compensate for minor differences in frame rates and fluctuations in USB / CSI link latency, achieving hardware and software coordination and synchronization.

[0104] In this embodiment, preferably, in step S1, an image quality assessment module is added to the acquisition thread to perform motion blur detection (based on gradient variance) and exposure anomaly detection (based on histogram distribution) on the frames, automatically removing obviously blurred or overexposed / underexposed frames, reducing the amount of invalid data in subsequent processing, and improving the overall detection accuracy and real-time performance.

[0105] like Figure 4 As shown, through the dual means of hardware synchronization and software optimization, the first step not only ensures the temporal consistency and spatial registration accuracy of RGB and IR video frames, but also improves the quality and information complementarity of input data at the source, laying a stable and reliable foundation for subsequent multimodal feature fusion and target detection.

[0106] Specifically, in this embodiment, a binocular camera system is installed in the vehicle's cockpit environment, including a visible light camera (RGB, resolution 1920×1080, frame rate 30 FPS) and a near-infrared camera (IR, resolution 1920×1080, frame rate 30 FPS). The two cameras are calibrated to obtain the intrinsic parameter matrix K, the extrinsic parameter rotation matrix R, and the translation vector t. Their installation positions need to ensure that their fields of view overlap and cover the driver's upper body area. In this embodiment, it is fixed at the left A-pillar.

[0107] Specifically, in this embodiment, a time synchronization module is deployed at the software acquisition end: an independent circular buffer queue is constructed to cache RGB frames and IR frames respectively; a unified high-precision timestamp is used for recording (precision 1 ms); and the frame alignment tolerance is set to not exceed ±1 / 2 frame period (approximately 16.7 ms).

[0108] In this embodiment, the quality detection module specifically performs the following detections on each frame of data:

[0109] (1) Motion blur detection: Based on gradient variance. In this embodiment, the gradient variance threshold is... Set as .

[0110] (2) Exposure anomaly detection: Based on image histogram distribution. In this embodiment, the histogram balance range is required to be... .

[0111] Frames outside the acceptable range are deemed invalid and automatically discarded. This embodiment achieves synchronized and high-quality data acquisition, providing stable input for subsequent detection.

[0112] Step S2: Perform a first-stage coarse detection on the RGB video frames to generate an attention heatmap, extract the region of interest (ROI), and perform feature enhancement and background suppression on the ROI to obtain a weighted RGB image.

[0113] In this embodiment, step S2 preferably includes the following steps:

[0114] Step S21: Synchronize the RGB video frames Input to a trained lightweight ResNet network or an equivalent convolutional neural network Extract its multi-channel intermediate feature map. , , Then, the attention heatmap is obtained by dimensionality reduction through 1×1 convolution and concatenation with the Sigmoid activation map:

[0115]

[0116] in, This indicates the probability that the corresponding pixel belongs to the target object (such as a mobile phone, cigarette, or seat belt). For activation function, For a 1×1 convolution kernel in the th... The weight parameters corresponding to each channel For the input feature map at the th Each channel, location eigenvalues ​​at that location For bias terms, The range of values ​​is .

[0117] Step S22: After obtaining the heat map, set a threshold. Binary segmentation is performed on the thermal values. The range of values ​​is Obtain the mask of the potential target area. :

[0118]

[0119] Step S23: Extract candidate region set through connected component analysis Each candidate region is defined as a Region of Interest (ROI) in the form of a minimum bounding rectangle. To enhance the features of the candidate regions and suppress background interference in subsequent detection, the set of ROIs is mapped to an attention weight mask. :

[0120]

[0121] in, The enhancement factor for the ROI region. The suppression coefficient for the background region;

[0122] Step S24: Apply the weight mask Applying the weighted method point by point to the original RGB image yields the weighted image. ,like Figure 5 As shown:

[0123]

[0124] In this embodiment, the ROI region weighting coefficient table is as follows:

[0125]

[0126] The advantages of the first-stage coarse detection and attention region generation method in this embodiment compared with the traditional full-image direct detection are shown in the table below:

[0127]

[0128] As can be seen, after the processing in step S2, the feature intensity of the target region is improved and the background region features are effectively suppressed. Thus, useless regions are efficiently eliminated in the first stage, and computing resources are concentrated on potential targets. While ensuring a high recall rate, the computing load and false detection probability of the subsequent fine detection stage are reduced.

[0129] Step S3: Perform geometric registration between the weighted RGB image and the synchronized IR image, and then perform multimodal fusion between the weighted RGB image and the registered IR image. Select the fusion path based on the ambient light index to form a multi-channel input, as detailed below. Figure 6 As shown.

[0130] In this embodiment, geometric registration is specifically performed on pose and scale to ensure that corresponding pixel positions point to the same 3D scene position in both images. Geometric registration can be performed using perspective transformation with calibration parameters (intrinsic and extrinsic parameters), as follows:

[0131]

[0132] in, Pixel position in the original infrared (IR) image grayscale values ​​or pixel vectors, Let K denote the projection transformation function, where K is the intrinsic parameter matrix, and R and t are the rotation and translation matrices, respectively.

[0133] In this embodiment, preferably, in step S3, the registered RGB image With IR plot Multimodal fusion can be performed using the following two strategies:

[0134] Path A fusion (RGB-dominant): The attention-weighted RGB three-channel image and the registered IR single-channel image are directly stacked along the channel dimension. The resulting H×W×4 multi-channel input is fed into the YOLO backbone network.

[0135] Path B fusion (IR-dominant): In poor lighting conditions, the IR image can be expanded into a three-channel base map, and then combined with the RGB single-channel luminance map in the channel dimension. Stacking creates another four-channel input:

[0136] This bidirectional fusion strategy can dynamically select or simultaneously compute two input paths based on the lighting environment inside and outside the vehicle, thereby improving robustness.

[0137] Illumination-adaptive path selection: Real-time calculation of the current ambient light index :

[0138]

[0139] in, This is the average RGB brightness (usually derived from camera sensor data, typically ranging from 0 to 255; normalization to 0 to 1 is recommended). Infrared contrast (IR Contrast, which reflects the effectiveness of infrared supplementary lighting, is usually obtained through infrared sensors or image analysis, and it is recommended to normalize the range to 0~1). The average RGB brightness weight reflects the "sufficiency of visible light". Infrared contrast weights reflect "infrared confidence level". In this embodiment, , .

[0140] when Choose path A at this time;

[0141] when Choose path B;

[0142] when At that time, path A and path B are computed in parallel;

[0143] in, This is the high light threshold; anything above it is considered "sufficient visible light." In this embodiment, , In this embodiment, as a low-light threshold, A light index below this level is considered "insufficient visible light." Therefore, the ambient light index... The closer the index is to 1, the more sufficient the ambient light is, and infrared supplementary lighting is not required; Ambient Light Index The closer it is to 0, the weaker the ambient light is, and infrared supplemental lighting is required.

[0144] Step S4: Input the fused image into the improved YOLO detection network for the second stage of fine detection, and then weight and fuse the output confidence of the YOLO detection network with the statistical scores of the heatmap region in the first stage to obtain the final detection result.

[0145] In this embodiment, preferably, in step S4, the detection network uses an improved YOLO series single-stage detector, and the inference layer structure is optimized for in-vehicle scenarios and multimodal inputs:

[0146] At the model structure level, the input convolutional kernels of the YOLO backbone network were adjusted to adapt... Channel input:

[0147]

[0148] Simultaneously, a cross-scale feature fusion path is introduced into the Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) structures:

[0149]

[0150] in, Represents the set of adjacent scales. For upsampling / downsampling alignment operation, Learnable weights are used to enhance the multi-resolution representation of features of small targets (such as cigarettes and mobile phones), such as... Figure 7 As shown.

[0151] In this embodiment, preferably, in step S4, the K-means++ clustering algorithm is used to optimize the anchor size to match the scale of small targets inside the vehicle. Specifically, the K-means++ clustering algorithm is used to optimize the anchor size in the training set of labeled boxes. Minimize the average Distance, to obtain the optimal set of Anchor sizes The optimization objective is:

[0152]

[0153] Among them, intersection and union ratio The calculation formula is: This optimization ensures that the Anchor is highly matched to the scale of the real target inside the vehicle, thereby significantly reducing the bounding box localization error.

[0154] In this embodiment, specifically in step S4, during the prediction phase, the detection head at each scale outputs a vector for each Anchor: ,in, For bounding box regression parameters, The probability of the existence of the target. For category The conditional probability. Decoded into the final set using the YOLO model. ,in, This refers to the horizontal / vertical pixel coordinates of the detection box in the input image coordinate system, as well as the width / height of the detection box. Number the target category. This represents the final confidence level for that category.

[0155] Meanwhile, at the software inference level, the system implements dynamic scheduling and result optimization mechanisms. The software detection engine determines the results based on the number of ROIs in the first stage. Based on density, it automatically switches between full-image mode and ROI cropping mode:

[0156]

[0157] in, This represents the number of effective ROIs generated by the first-stage attention heatmap. For mode switching threshold, The range of values ​​is Specifically, in this embodiment, .when No more than When ROI is considered sparse, then it is adopted Cutting and pasting reasoning; when Higher than In this case, the ROI is considered dense, and full-graph inference is performed directly to avoid the overhead of repeated pruning.

[0158] On-demand inference is achieved, significantly reducing unnecessary computation. In terms of runtime architecture, video acquisition, geometric registration, feature fusion, and YOLO inference are divided into parallel threads, and data is transferred through a circular buffer queue to achieve pipelined processing throughout the entire process, thereby reducing latency and improving real-time performance.

[0159] In this embodiment, preferably, in step S4, during the detection result fusion stage, the software layer will output the confidence level of YOLO. Compared with the first stage heatmap score Double weighting:

[0160]

[0161] in, To determine the final confidence score, The target confidence score output by the YOLO master detection network. This represents the statistical score of the first-stage attention heatmap within the detection bounding box region. For dynamic adjustment coefficients, It can be dynamically adjusted according to factors such as ambient light and occlusion rate to ensure the robustness of the test results. The range of values ​​is The final detection results will also enter a cache pool for time consistency optimization, eliminating false detections in single frames and compensating for short-term missed detections. The overall flowchart in this embodiment is as follows: Figure 8 As shown.

[0162] By combining the above-mentioned model structure optimization with the software intelligent scheduling algorithm, this module significantly improves the detection accuracy and stability of various in-vehicle targets, including faces, hands, mobile phones, cigarettes, and seat belt buckles, while maintaining the real-time operation of automotive-grade hardware. It also has strong robustness to complex working conditions such as low light, strong light, and partial occlusion.

[0163] In this embodiment, preferably, the training method for the attention network used in the first stage coarse detection in step S2 includes:

[0164] Step 1: Build a training dataset and generate GT (Ground Truth) heatmap labels using professional annotation tools.

[0165] In this embodiment, preferably, in step one, the training dataset includes publicly available driver behavior detection datasets (such as Drive&Act, AUC Distracted Driver, SafeDrive, etc.) and self-collected in-vehicle multi-environment video frames. The self-collected video frames cover strong light, weak light, backlight conditions and behavior categories such as making phone calls, smoking, looking down, and drinking water.

[0166] Step 2: Label the target region ROI for each frame in the training dataset and convert the ROI into a probabilistic two-dimensional Gaussian distribution heatmap to guide the attention network to learn the spatial distribution of possible target regions.

[0167] Step 3: Use the Binary Cross Entropy (BCE) loss to calculate the predicted heatmap. Compared with the actual heat map Differences between them:

[0168] ;

[0169] Where N is the total number of pixels in the heatmap. is the BCE loss value, which can maintain high sensitivity even when the ratio of positive to negative samples is unbalanced, which is beneficial to improving high recall. i is the pixel index.

[0170] In this embodiment, preferably, in step S4, during the second stage of YOLO main detection network training, the input data is the fused form after the first stage processing. This can be either a fused RGB and IR four-channel image, or a feature tensor obtained by feature-level fusion of RGB and IR features obtained separately from the feature extraction network in an intermediate layer. The training and optimization strategy for the second stage of the YOLO main detection network includes:

[0171] A multimodal joint training strategy is adopted, which allows RGB and IR to be input into the detection network simultaneously in the same batch. The information of both modalities is retained throughout the entire process from feature extraction to detection prediction, thereby improving the robustness of the network under low light, strong light, complex background and local occlusion conditions.

[0172] To address the high risk of missed detection for small objects inside vehicles (such as cigarettes and mobile phones), Focla Loss is introduced in the classification branch to give higher weights to samples that are more difficult to detect.

[0173]

[0174] in, The Focla Loss value is... This is a class balancing factor used to assign different weights to positive and negative samples in an imbalanced dataset. To focus on the parameters, in this embodiment Values , The difficulty level of the sample is adjusted by a factor that, when close to 0, reduces the sample's contribution to the loss, while a factor that, when close to 1, retains a high loss weight.

[0175] In the high-resolution feature layer where small targets frequently appear, denser small-sized anchors are added, and a spatial attention mechanism is incorporated into this feature layer to enhance the feature representation of small targets.

[0176] The optimization process employs a training strategy based on SGD, combined with cosine annealing learning rate scheduling and a warm-up phase, enabling the network to converge smoothly in the early stages and making the gradients more adaptive in the later stages.

[0177] Simultaneously, mixed precision training (FP16) is used to improve computational efficiency and memory utilization, thereby ensuring a significant improvement in detection accuracy, especially for small targets, while maintaining the inference speed within the range that automotive-grade embedded platforms can run in real time.

[0178] The in-vehicle target detection method based on binocular multimodal fusion and two-stage attention mechanism in this embodiment was tested with existing models on different training sets, and the performance comparison is shown in the table below:

[0179]

[0180] As can be seen from the mAP@0.5 metric, the proposed solution achieved significant advantages on all three datasets, with an average improvement of 5-7 percentage points compared to the original YOLOv8, and reaching a maximum of 96.3% on the self-collected multi-environment dataset. Regarding the recall rate of small targets, the proposed model showed a particularly significant improvement compared to the original YOLOv8, with an average improvement of 7%-12%, effectively improving the problem of missed detections of small targets such as cigarettes and mobile phones.

[0181] Regarding the false detection rate, this invention, through a first-stage ROI background suppression and a second-stage dual-weighted fusion mechanism, stably reduces the false detection rate to the range of 3%–4%, approximately half that of the original YOLOv8. Latency and frame rate metrics show that this invention, utilizing an on-demand inference dynamic scheduling mechanism, reduces the average inference latency by 46 ms and increases the FPS to over 40 frames per second while maintaining detection accuracy, meeting the real-time operation requirements of automotive-grade embedded platforms.

[0182] Furthermore, the present invention exhibits high robustness in various environments, including low light, strong backlight, and complex in-vehicle backgrounds, and its performance fluctuations across public and self-collected datasets are minimal, verifying the wide adaptability and significant effect of the training optimization strategy (BCE+Focal Loss, multimodal joint training, and small target feature enhancement) in practical deployment scenarios.

[0183] The in-vehicle target detection method in this embodiment is based on a binocular multimodal input illumination adaptive fusion mechanism. It utilizes RGB and IR binocular cameras to simultaneously acquire data, and geometric registration ensures pixel-level alignment. It provides two fusion modes: path A (RGB main) and path B (IR main), and calculates the current ambient lighting conditions in real time using the illumination index to achieve dynamic path selection or parallel fusion. It maintains stable detection performance under various lighting environments such as strong light, weak light, and backlight, significantly enhancing system robustness and avoiding detection failure of single-modality under certain lighting conditions.

[0184] This embodiment's in-vehicle target detection method is based on a two-stage detection architecture: first-stage attention heatmap ROI enhancement and second-stage improved YOLO. The first stage generates a probabilistic heatmap using a lightweight network, and generates a weighted image based on the ROI enhancement coefficient α and background suppression coefficient β to enhance potential target features and suppress background interference. The second stage introduces multi-channel Cf input adaptation, cross-scale learnable feature fusion (FPN+PAN enhancement), and K-means++ optimized anchors into the improved YOLO detection network to improve the detection accuracy of small targets (such as cigarettes, mobile phones, and seatbelt buckles). While maintaining real-time performance, it significantly improves the recall rate and overall detection accuracy of small targets and reduces the false detection rate.

[0185] The in-vehicle target detection method in this embodiment is based on dynamic inference scheduling and a multi-level result optimization mechanism. It dynamically switches between full-image mode and ROI clipping mode according to the number and density of ROIs in the first stage, reducing the computational burden on irrelevant areas. In the detection stage, the YOLO output confidence score and the first-stage ROI thermal score are weighted twice (the fusion coefficient γ can be adaptively adjusted according to lighting, occlusion, etc.), and a time-consistent caching mechanism is used to eliminate false detections in single frames and compensate for short-term missed detections. This reduces the measured inference latency (s) and improves the running frame rate.

[0186] Example 2

[0187] The in-vehicle target detection system based on binocular multimodal fusion and two-stage attention mechanism in this embodiment adopts the in-vehicle target detection method based on binocular multimodal fusion and two-stage attention mechanism in Embodiment 1, and specifically includes the following modules:

[0188] The binocular acquisition module includes a visible light camera and an infrared camera, used to simultaneously acquire RGB video streams and IR video streams inside the vehicle.

[0189] The synchronization optimization module includes a circular buffer unit and a quality evaluation unit. The circular buffer unit is used to cache RGB frames and IR frames and achieve timing synchronization through timestamp matching. The quality evaluation unit is used to detect motion blur and exposure abnormalities in video frames and remove invalid frames.

[0190] The attention generation module is used to input synchronized RGB video frames into a lightweight ResNet network, generate attention heatmaps and attention weight masks, and perform weighted processing on the original RGB images.

[0191] The multimodal fusion module is used to perform geometric registration on weighted RGB images and IR images, calculate the ambient illumination index, select or execute the corresponding feature fusion path in parallel, and output multi-channel fused data.

[0192] The target detection module includes an improved YOLO detection network, an inference scheduling unit, and a fusion optimization unit. The improved YOLO detection network is used to perform target detection on multi-channel fused data. The inference scheduling unit is used to dynamically switch detection modes. The fusion optimization unit is used to perform weighted fusion and time consistency optimization on the detection results and output the final detection results.

[0193] Many specific details have been set forth in the foregoing description to provide a thorough understanding of the present invention. However, the above description is merely a preferred embodiment of the present invention, and the present invention can be implemented in many other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed above. Furthermore, any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention, or modify them into equivalent embodiments, using the methods and techniques disclosed above, without departing from the scope of the present invention. Any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A method for in-vehicle target detection based on binocular multi-modal fusion and two-stage attention mechanism, characterized in that: Comprise the following steps: Step S1: Obtain the RGB video stream and the IR video stream through the visible light camera and the infrared camera, the exposure time of the two cameras is synchronized with the frame rate, and a unified timestamp mechanism is used at the acquisition end to match and align the acquired RGB and IR frames one by one; Step S2: Perform first-stage coarse detection on the RGB video frame to generate an attention heat map, extract a region of interest ROI, and perform feature enhancement and background suppression on the ROI region to obtain a weighted processed RGB image; Step S3: Perform geometric registration on the weighted processed RGB image and the synchronous IR image, and perform multi-modal fusion on the weighted processed RGB image and the registered IR image, select a fusion path according to the ambient light index, and form a multi-channel input; Step S4: Input the fused image into the improved YOLO detection network for second-stage fine detection, and weight fuse the output confidence of the YOLO detection network and the statistical score of the first-stage heat map region to obtain the final detection result.

2. The in-vehicle target detection method based on binocular multi-modal fusion and two-stage attention mechanism according to claim 1, characterized in that: In step S1, a ring buffer queue is realized at the acquisition end to buffer the RGB and IR video frames respectively, and the frame alignment threshold is dynamically adjusted through a timestamp matching algorithm to ensure that the output frame pair can be transmitted to the downstream processing module within a synchronization error range of ±1 / 2 frame period.

3. The in-vehicle target detection method based on binocular multi-modal fusion and two-stage attention mechanism according to claim 1, characterized in that: In step S1, an image quality evaluation module is added in the acquisition thread to perform motion blur detection and exposure abnormality detection on the frame, and automatically remove obviously blurred or overexposed / underexposed frames.

4. The method of claim 1, wherein the method is characterized by: In step S2, the following steps are specifically included: Step S21: inputting the time-synchronized RGB video frame to the trained lightweight ResNet network or equivalent convolutional neural network , extracting its multi-channel intermediate feature map, , , and then obtaining the attention heat map through 1x1 convolution dimension reduction and connection of the Sigmoid activation mapping. in, This indicates the probability that the corresponding pixel belongs to the target object. For activation function, For a 1×1 convolution kernel in the th... The weight parameters corresponding to each channel For the input feature map at the th Each channel, location eigenvalues ​​at that location For bias terms, The range of values ​​is ; Step S22: After obtaining the heat map, a threshold is set , a binary segmentation is performed on the heat value, , the value range of the threshold is , a potential target region mask is obtained : Step S23: Extracting a candidate region set through connected domain analysis Each candidate region is defined as a region of interest (ROI) in the form of a minimum circumscribed rectangle. The ROI set is mapped to an attention weight mask : wherein, is an enhancement coefficient for the ROI region, is an inhibition coefficient for the background region; Step S24: applying the weight mask to the original RGB image point-wise operation on the original RGB image to obtain a weighted image :

5. The method of claim 1, wherein the method is characterized by: In step S3, the registered RGB image with the IR image The multimodal fusion is performed by the following two strategies: Path A fusion: the RGB three-channel image weighted by attention and the registered IR single-channel image are directly stacked in the channel dimension: The HxWx4 multi-channel input is obtained and sent to the YOLO backbone network; Path B fusion: extend the IR image to a three-channel base image, and then concatenate with the RGB single-channel luminance image in the channel dimension Stacking, forming another four-channel input: Real-time calculation of current ambient light index : wherein, is the RGB average luminance, is the infrared contrast, is the RGB average luminance weight, is the infrared contrast weight, ; When path A is selected; When path B is selected; When paths A and B are computed in parallel; wherein, is a high light threshold, above which it is considered that "sufficient visible light", is a low light threshold, below which it is considered that "insufficient visible light".

6. The method of claim 1, wherein the method is characterized by: In step S4, the detection network uses an improved YOLO series single-stage detector: At the model structure level, the input convolution kernel of the YOLO backbone network is adjusted to adapt to Channel input: Cross-scale feature fusion paths are introduced in the feature pyramid network FPN and the path aggregation network PAN structure: wherein, denotes a set of adjacent scales, is an up / down sampling alignment operation, is a learnable weight for enhancing the multi-resolution representation of small target features.

7. The method of claim 1, wherein the method is characterized by: In step S4, the K-means++ clustering algorithm is used to optimize the Anchor size, match the small target scale in the vehicle, and specifically: through the K-means++ clustering algorithm, the training set label box set is clustered to obtain the optimal Anchor size set The average Distance is minimized The optimization target is: The formula for calculating the intersection-over-union is: ​ 8. The method of claim 1, wherein the method is characterized by: In step S4, in the detection result fusion link, the software layer will double-weight the YOLO output confidence with the first stage heat map score perform wherein, is the final detection confidence score, is the target confidence score output by the YOLO main detection network, is the statistical score of the first-stage attention heat map in the detection frame region, is a dynamic adjustment coefficient, the value range of is .

9. The method of claim 1, wherein the method is characterized by: The training method of the attention network used in the first-stage coarse detection in step S2 includes: Step one: build a training data set, and generate GT heat map labels through a professional labeling tool; Step two: label the target region ROI in each frame of the training data set, and convert the ROI into a probability type two-dimensional Gaussian distribution heat map to guide the attention network to learn the spatial distribution of the target possible region; Step three: Calculate the predicted heat map using Binary Cross Entropy (BCE) loss The difference between the real heat map and the predicted heat map: ； where N is the total number of pixels of the thermal image, is the BCE loss value, and i is the pixel index.

10. The method of claim 9, wherein the method is characterized by: In step one, the training data set includes a public driver behavior detection data set and self-collected in-vehicle multi-environment video frames, and the self-collected video frames cover strong light, weak light, backlight lighting conditions, and behaviors such as making a phone call, smoking, looking down, and drinking water.

11. The method of claim 1, wherein the method is characterized by: In step S4, the training and optimization strategy of the YOLO main detection network in the second stage includes: Adopt a multi-modal joint training strategy to input RGB and IR into the detection network in the same batch, and retain the information of the two modalities from feature extraction to detection prediction; Introduce Focla Loss in the classification branch to give higher weight to samples with greater detection difficulty: wherein, is a Focal Loss loss value, is a class balancing factor to assign different weights to positive and negative samples in a class imbalanced dataset, is a focal parameter, is a difficulty sample adjustment factor, which factor close to 0 reduces its loss contribution, which factor close to 1 retains high loss weight; Increase more dense small size Anchors in the high resolution feature layer where small targets often appear, and integrate a spatial attention mechanism into the feature layer to enhance the small target feature expression ability; The training strategy based on SGD is adopted in the optimization process, combined with cosine annealing learning rate scheduling and preheating stage, so that the network converges smoothly in the early stage and the gradient is more adaptive in the later stage.

12. An in-vehicle target detection system based on binocular multi-modal fusion and two-stage attention mechanism, characterized in that: The binocular multi-modal fusion and two-stage attention mechanism-based in-vehicle target detection method according to any one of claims 1-11, specifically comprising the following modules: The binocular acquisition module includes a visible light camera and an infrared camera, which are used to synchronously acquire RGB video streams and IR video streams in the vehicle; The synchronous optimization module includes a ring buffer unit and a quality evaluation unit, the ring buffer unit is used to cache RGB frames and IR frames and realize time synchronization through timestamp matching, and the quality evaluation unit is used to detect motion blur and exposure abnormalities of the video frames and eliminate invalid frames; The attention generation module is used to input the synchronized RGB video frames into a lightweight ResNet network, generate attention heat maps and attention weight masks, and perform weighted processing on the original RGB images; The multi-modal fusion module is used to perform geometric registration on the weighted RGB images and IR images, calculate the ambient light index, and select or perform corresponding feature fusion paths in parallel to output multi-channel fusion data; The target detection module includes an improved YOLO detection network, an inference scheduling unit and a fusion optimization unit, the improved YOLO detection network is used to detect targets in the multi-channel fusion data, the inference scheduling unit is used to dynamically switch the detection mode, and the fusion optimization unit is used to perform weighted fusion and time consistency optimization on the detection results and output the final detection results.

Images