Visual detection method for foreign matters and dangerous driving behaviors in cab

Abstract

The invention relates to the technical field of intelligent cabins and driving safety, and discloses a visual detection method for foreign matters in a cab and dangerous driving behaviors, which comprises the following steps: acquiring image data of the cab and real-time speed of a vehicle, identifying the dangerous foreign matters in a safety airbag deployment area based on the image data, and determining material attributes; identifying a dangerous driving behavior of the passenger and determining the type of the dangerous driving behavior; calculating a basic risk score according to the foreign matter material and the behavior type; determining a speed regulation factor according to the real-time vehicle speed; performing operation on the basic risk score and the speed regulation factor to obtain a comprehensive risk score; and mapping the comprehensive risk score into a preset danger level signal, and executing corresponding early warning and intervention measures according to the signal. According to the method, the static in-cabin risk is combined with the dynamic driving working condition, the graded response strategy is realized, the comprehensive risk in the cab can be quantitatively evaluated, and the effectiveness and intelligent level of early warning and intervention are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit and driving safety technology, specifically to a visual detection method for foreign objects in the driver's cab and dangerous driving behaviors. Background Technology

[0002] With the continuous improvement of automotive intelligence, driving safety and cabin experience have become important focuses in vehicle design and development. In order to achieve intelligent perception and active safety protection of the driver's cabin environment, visual monitoring systems, especially camera-based driver and passenger monitoring systems, have been applied in vehicles. These systems can achieve functions such as driver fatigue detection, attention distraction reminders, and passenger position recognition by analyzing images in the driver's cabin, providing technical support for improving driving safety.

[0003] However, existing cockpit visual monitoring technologies have limitations in the depth and breadth of risk assessment. On the one hand, most systems focus either on detecting the presence of foreign objects or on judging the posture of occupants, lacking the comprehensive analytical capability to combine both. After identifying risk factors, the system cannot perform in-depth attribute analysis. For example, the system can identify the presence of an object in the airbag area, but cannot distinguish whether the object is a soft toy or a hard metal cup, thus failing to differentiate the distinct collision risks represented by the two. On the other hand, the system's risk assessment is based on static, isolated snapshots of the cabin state, ignoring the influence of the vehicle itself as a dynamic system. For instance, the threat posed by a dangerous driving posture when the vehicle is stationary or traveling at low speed differs from the threat posed at high speed.

[0004] Existing technologies fail to integrate static risk information within the cabin with the vehicle's real-time driving status, resulting in a disconnect between risk assessment results and actual driving conditions. Consequently, their output warning signals are fixed and cannot be dynamically adjusted based on the actual urgency of the risk, thus reducing the effectiveness and reliability of the safety system. Therefore, this invention proposes a visual detection method for foreign objects in the driver's cab and dangerous driving behaviors to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a visual detection method for foreign objects in the driver's cab and dangerous driving behaviors, solving the problem that inaccurate risk assessment and the lack of hierarchical and dynamic adjustment capabilities in early warning and intervention measures make it difficult to adapt to real driving scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a visual detection method for foreign objects in the driver's cab and dangerous driving behaviors, comprising:

[0007] First, image data from inside the driver's cab and the vehicle's real-time speed are acquired. Based on the acquired image data, two recognition tasks are performed in parallel. One task is to identify dangerous foreign objects located within the preset airbag deployment area and determine the material properties of the dangerous foreign objects. The other task is to identify dangerous driving behaviors of passengers and determine the type of dangerous driving behaviors.

[0008] Then, a basic risk score is calculated based on the material properties of the identified dangerous foreign object and the type of the identified dangerous driving behavior. At the same time, a speed adjustment factor is determined based on the real-time vehicle speed. The calculated basic risk score is then multiplied by the determined speed adjustment factor to obtain a comprehensive risk score.

[0009] Finally, the obtained comprehensive risk score is mapped to a preset hazard level signal, and based on the mapped hazard level signal, corresponding early warning and intervention measures are implemented.

[0010] Preferably, the step of identifying dangerous foreign objects and determining their material properties specifically involves: processing the acquired image data using a single-stage target detection network architecture to output candidate bounding boxes containing the location of the foreign object;

[0011] The image regions within the output candidate bounding boxes are processed by independent cascaded risk attribute classifiers to determine the material properties of hazardous foreign objects.

[0012] Preferably, the step of identifying dangerous driving behaviors of passengers and determining the type of dangerous driving behaviors specifically involves: processing the acquired image data through a bottom-up attitude estimation submodule to obtain a set of skeletal key point coordinates of the passenger;

[0013] Based on the acquired set of skeletal key point coordinates, a set of rule-based judgment logic based on geometric constraints is executed to identify predefined dangerous driving behaviors.

[0014] Preferably, the step of calculating the basic risk score specifically involves: setting basic risk weights for each predefined type of dangerous driving behavior and each material attribute of a dangerous foreign object;

[0015] The basic risk score is obtained by summing the basic risk weights corresponding to all dangerous driving behaviors and dangerous foreign objects identified at the current moment.

[0016] Preferably, when calculating the basic risk score, if the identified dangerous foreign object is located in the steering wheel area, a preset regional risk gain weight is added to the dangerous foreign object during the accumulation of basic risk weights.

[0017] Preferably, the step of determining the speed adjustment factor specifically involves comparing the acquired real-time vehicle speed with at least one preset speed threshold.

[0018] Based on the comparison results, the speed adjustment factor was selected from a number of preset risk adjustment coefficients that monotonically increase with the speed range.

[0019] Preferably, the step of mapping the obtained comprehensive risk score to a preset danger level signal specifically involves: comparing the obtained comprehensive risk score with a set of preset, strictly monotonically increasing numerical thresholds to determine the numerical range in which the comprehensive risk score lies;

[0020] Based on the numerical range of the comprehensive risk score, the corresponding discrete hazard level is determined as the hazard level signal.

[0021] Preferably, the set of rule-based judgment logic based on geometric constraints includes: comparing the coordinates of the passenger's knee and ankle key points in the acquired set of skeletal key point coordinates with the spatial position of a predefined danger zone to determine whether there is a dangerous driving behavior caused by the passenger's lower limbs entering the predefined danger zone.

[0022] Preferably, when the mapped danger level signal is low risk or medium risk, the warning and intervention measures are to execute a visual warning, which includes rendering a highlighted bounding box of the identified dangerous foreign object in the real-time video image of the central control display screen.

[0023] Preferably, when the mapped danger level signal is high risk or extremely high risk, the warning and intervention measures are to perform vehicle intervention, which includes sending a command to the body control module to turn on the hazard warning lights and sending a command to the seat belt control module to trigger the seat belt pretensioner for point-controlled tightening.

[0024] The above solution achieves the following beneficial technical effects:

[0025] 1. This invention quantifies the identified dangerous foreign object materials and dangerous driving behavior types into basic risk scores, and introduces a speed adjustment factor determined based on real-time vehicle speed to weight the basic risk scores. This method can integrate static cabin risks with dynamic driving risks, so that the same dangerous behavior generates a higher risk score at high speeds than at low speeds, thereby making risk assessment more accurate and dynamic.

[0026] 2. This invention achieves simultaneous monitoring of multi-source risks by identifying dangerous foreign objects and dangerous driving behaviors in parallel. Furthermore, this method not only identifies the presence of foreign objects but also determines their material properties; it not only identifies improper postures but also determines their specific behavioral types. This in-depth analysis enables the system to distinguish the distinctly different threat levels represented by soft toys and metal water cups in the airbag area, thus providing a foundation for subsequent risk quantification.

[0027] 3. This invention maps the final calculated comprehensive risk score into discrete hazard level signals. This method can trigger different response measures according to different hazard levels. For low and medium risks, a reminder-style visual warning is used; for high and extremely high risks, it is upgraded to a strongly interventionist vehicle control command. This response mechanism that matches the risk level ensures that sufficiently strong measures can be taken to ensure safety at critical moments, and also avoids unnecessary interference to users due to excessive intervention in low-risk scenarios. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the overall system architecture and method of the present invention;

[0029] Figure 2 This is a schematic diagram illustrating the risk source identification of the present invention;

[0030] Figure 3 This is a schematic diagram of the dynamic risk assessment and graded early warning strategy of the present invention.

[0031] The module includes: 10, image acquisition unit; 20, central processing unit; 30, vehicle data interface unit; 40, human-machine interaction and execution unit; 21, data-driven and preprocessing module; 22, parallel risk source identification module; 23, dynamic risk assessment and decision-making module; and 24, early warning and intervention control module. Detailed Implementation

[0032] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] See attached document Figure 1 The present invention provides a visual detection method for foreign objects in the driver's cab and dangerous driving behaviors, the execution of which can rely on a set of hardware units deployed inside the vehicle.

[0034] The method involves using an image acquisition unit 10 to acquire image data from the driver's cab. The image acquisition unit 10 is installed in the central area of ​​the front roof of the vehicle, for example, integrated into the rearview mirror base or the roof console assembly. This location ensures that the field of view of the image acquisition unit 10 can completely cover the area where the steering wheel is located, the airbag cover area on the passenger side center console, and the main activity space of the upper body and legs of the passenger.

[0035] The image acquisition unit 10 physically includes a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor with a resolution of 1920x1080 pixels or higher and an image acquisition frame rate set to no less than 15 frames per second. This sensor features wide dynamic range (WDR) functionality to compensate for exposure in highlight and shadow areas within the driver's cab, acquiring image data with balanced detail. Working in conjunction with the sensor is a wide-angle lens, whose focal length is set to ensure a horizontal field of view of no less than 120 degrees at a preset installation height.

[0036] Before performing the image data processing steps of this method, the image acquisition unit 10 needs to be calibrated to obtain its internal parameters and correct lens distortion, and the points in the image coordinate system need to be calibrated. 3D points in the camera coordinate system The conversion relationship between them follows the pinhole camera model:

[0037] ;

[0038] in, Scale factor; These are pixel coordinates; Principal point coordinates; and This represents the normalized focal length of the camera along the x and y axes.

[0039] The following model is used to correct the radial and tangential distortions produced by wide-angle lenses, assuming the coordinates of an undistorted image point are... Its corresponding distorted coordinates are The correction process is as follows:

[0040] ;

[0041] ;

[0042] in, ; Radial distortion coefficient; These calibration parameters, which are tangential distortion coefficients, are stored and used to preprocess each subsequent frame of image data.

[0043] The algorithm model operation steps of this method are executed by a central processing unit 20, which is integrated in the vehicle's intelligent cockpit domain controller. Its physical form is a system-on-a-chip (SoC) containing a dedicated computing core. The SoC integrates a neural network processing unit (NPU) or a graphics processing unit (GPU) to provide hardware acceleration for the inference calculation of the deep learning model involved in this method.

[0044] The execution of this method also requires the acquisition of real-time dynamic data of the vehicle. This data is obtained through a vehicle data interface unit 30. The vehicle data interface unit 30 is physically a controller local area network (CAN) bus transceiver, which is connected to the CAN bus of the vehicle chassis or body. The vehicle data interface unit 30 is responsible for listening to the data frames on the bus and parsing out the data field containing real-time vehicle speed information from them, and then providing the vehicle speed data to the central processing unit 20.

[0045] The warning and intervention steps of this method are implemented through a human-machine interaction and execution unit 40, which consists of multiple distributed components, including: an LCD screen located on the center console, an instrument panel display screen in front of the driver, speakers of the vehicle audio system, and a controllable ambient lighting module integrated in the vehicle. These components receive instructions from the central processing unit 20. For intervention actions that require control of the vehicle's external lights (such as hazard warning lights), the central processing unit 20 sends control instructions to the body control module (BCM), which directly drives the corresponding headlight relays or controllers.

[0046] See attached document Figure 1 In order to perform the aforementioned visual detection method for foreign objects in the cab and dangerous driving behaviors, the memory of the central processing unit 20 stores computer-readable instructions, which are logically divided into multiple cooperative functional modules.

[0047] Data-driven and preprocessing module 21: The data-driven and preprocessing module 21 is configured to establish a communication connection with the image acquisition unit 10 and the vehicle data interface unit 30. The data-driven and preprocessing module 21 includes a video stream capture subroutine for reading the raw digital signal output by the image sensor in frames.

[0048] For each frame of raw image data acquired The data-driven and preprocessing module 21 performs size adjustment and pixel value normalization operations to generate tensor data that meets the input requirements of subsequent deep neural networks. The normalization process uses a linear mapping method to normalize each pixel in the image. In the Pixel grayscale values ​​on each channel Mapping to floating-point range or The specific normalization calculation formula is as follows:

[0049] ;

[0050] in, These are the normalized pixel values; For the preset first Channel pixel mean; For the preset first Channel pixel standard deviation, these parameters and The training dataset for the deep learning model is statistically obtained, and the data-driven and preprocessing module 21 outputs the preprocessed image tensor. .

[0051] Risk Source Parallel Identification Module 22: The risk source parallel identification module 22 is the core component for executing the visual perception algorithm. It receives the preprocessed image tensor. As input, the risk source parallel identification module 22 is logically further divided into two parallel execution sub-modules: foreign object detection sub-module and attitude estimation sub-module.

[0052] Foreign Object Detection Submodule: This submodule contains a convolutional neural network architecture for extracting spatial features from images. The convolutional neural network architecture performs convolution operations on the input tensor through a series of convolutional layers. Convolutional output feature map of the layer The calculation process is expressed as follows:

[0053] ;

[0054] in, For the first The first layer Each feature map; For the first layer above Each feature map; This is the corresponding convolution kernel matrix; Represents a two-dimensional convolution operation; For bias terms; It is a non-linear activation function.

[0055] The output layer of the foreign object detection submodule is configured to generate a list of foreign object detections. Each element in the foreign object detection list contains the following information:

[0056] ;

[0057] in, The coordinates of the rectangular bounding box surrounding the foreign object; This is a category index for foreign objects, corresponding to the aforementioned soft, solid, hard, or hard-fragile categories; This represents the confidence probability value of the detection result.

[0058] Attitude estimation submodule: The attitude estimation submodule contains a keypoint regression network for locating the anatomical keypoints of the front passenger. The keypoint regression network outputs a set of heatmaps, each corresponding to a specific human keypoint (e.g., the right knee). A key point, in image coordinates Response value at This indicates the probability that the keypoint is located at that position. The attitude estimation submodule determines the keypoint's coordinates by finding the maximum point on the heatmap. :

[0059] ;

[0060] The pose estimation submodule ultimately outputs a value containing... A set vector of coordinates of key points This is used for subsequent geometric rule determination.

[0061] Dynamic Risk Assessment and Decision Module 23: Dynamic Risk Assessment and Decision Module 23 is configured to receive a list of foreign object detections from Risk Source Parallel Identification Module 22. and key point set vector and real-time vehicle speed data from data-driven and preprocessing module 21. .

[0062] The dynamic risk assessment and decision-making module 23 internally maintains a state machine and a set of logical judgment rules. The module first... The bounding box coordinates in the mask determine whether the foreign object is located in the predefined airbag deployment area. Inside. The judgment condition is the center point of the bounding box. Intersection with the mask region:

[0063] ;

[0064] Subsequently, the dynamic risk assessment and decision-making module 23, based on Calculate the geometric relationship between the human skeleton vector and the vehicle space plane to identify dangerous postures.

[0065] The core function of the dynamic risk assessment and decision-making module 23 is to execute the risk level mapping function. This function outputs a discrete hazard level signal based on the input foreign object's material hardness, posture hazard level, and vehicle speed grading threshold. The signal takes values ​​from the set These correspond to five risk levels, ranging from extremely low to extremely high.

[0066] Early warning and intervention control module 24: Early warning and intervention control module 24 is configured to respond to input hazard level signals. It queries the preset policy mapping table and generates control instructions for different hardware devices.

[0067] The early warning and intervention control module 24 contains multiple instruction generators:

[0068] Display instruction generator: Generates image rendering instructions, overlays the foreign object bounding box and text warning layer onto the real-time video stream, and sends it to the central control display screen.

[0069] Audio command generator: Select the corresponding voice file or alarm waveform data and send it to the audio amplifier.

[0070] Vehicle control command generator: when At that time, a lighting control message conforming to the CAN bus protocol is generated. The lighting control message contains the control code for turning on the hazard lights and is sent to the body control module through the vehicle data interface unit 30.

[0071] See attached document Figure 1 The foreign object detection submodule is constructed using a single-stage target detection network architecture. This architecture simultaneously performs feature extraction, target localization, and category prediction in a single forward inference process. The network is logically divided into three parts: the backbone network, the neck network, and the detection head.

[0072] Backbone network:

[0073] The backbone network is configured to perform feature extraction tasks, taking a preprocessed RGB image tensor as input. The backbone network consists of stacked convolutional modules, each containing a 2D convolutional layer, a batch normalization layer, and a SiLU activation function. The mathematical expression of the SiLU activation function is:

[0074] ;

[0075] in, For input feature values, This is the standard Sigmoid function.

[0076] To enhance the network's ability to extract features from foreign objects of different scales, the backbone network adopts a cross-stage local network structure. The cross-stage local network structure divides the feature map of the basic feature layer into two parts in the channel dimension through a segmentation operation: one part is computed through dense residual blocks, and the other part is directly connected through skip connections. Finally, the parts are spliced ​​together in the channel dimension.

[0077] Neck network:

[0078] The neck network is located between the backbone network and the detection head. It is configured to perform multi-scale feature fusion. The multi-scale feature fusion adopts a structure that combines a feature pyramid network and a path aggregation network. The pyramid network layer passes high-level semantic features from top to bottom, and the path aggregation network layer passes low-level localization features from bottom to top.

[0079] In this process, for feature maps of two different scales and The fusion operation is achieved through upsampling and element-wise addition or channel concatenation, resulting in a fused feature map tensor. It can simultaneously retain semantic information that helps with classification and spatial information that helps with localization.

[0080] Detection head:

[0081] The detection head is configured to generate the final detection results. It contains three detection layers at different scales, corresponding to large, medium, and small target sizes, respectively. Each detection layer outputs a dimension... The tensor of, where, 3 represents the grid size of the feature map; 4 represents the number of anchor boxes preset for each grid cell; 5 represents the bounding box coordinate offset; 1 represents the target confidence level. Represents the probability of the category (in this embodiment) These correspond to the categories of soft, sturdy, hard, and hard-fragile, respectively.

[0082] For the bounding box coordinate parameters in the output tensor The network decodes it into predicted bounding boxes in the input image coordinate system. The decoding process follows the formula below:

[0083] ; ;

[0084] ; ;

[0085] in, , To predict the x and y coordinates of the bounding box center point in the feature map grid coordinate system; , To predict the width and height of the bounding box; , The offset coefficient of the center point of the bounding box predicted by the network relative to the top left corner of the grid cell; , The scaling factor for the width and height of the bounding box predicted by the network relative to the width and height of the anchor box; , The integer coordinates of the top-left corner of the current feature map grid cell; , Set the width and height of the preset anchor frame; This is the Sigmoid activation function, used to map predicted values ​​to a specific interval.

[0086] In addition, the detection head also outputs a target confidence score, which is a confidence parameter output by the network. After the Sigmoid function Therefore, during inference, the final confidence score for a particular class is obtained by multiplying the target confidence score by the conditional probability of that class.

[0087] ;

[0088] in, This represents the conditional class probability predicted by the network.

[0089] See attached document Figure 2 After the foreign object detection submodule completes target localization and outputs candidate bounding boxes, the cascaded risk attribute classifier is configured to perform secondary fine analysis on the image region within the bounding box to determine the physical material properties of the foreign object. The cascaded risk attribute classifier is an independent convolutional neural network, and its input is an image slice after the region of interest has been cropped and the size normalized.

[0090] The classifier first performs ROI alignment. Let the original image be a matrix. The bounding box coordinates output by the foreign object detection submodule are: The classifier from Extract the corresponding pixel matrix from the middle To accommodate the fixed input dimension of the classification network, the system uses bilinear interpolation to scale the extracted pixel matrix to a uniform size. (For example (pixels).

[0091] For any non-integer coordinate point in the target size image pixel value at The calculation formula is as follows:

[0092] ;

[0093] in, These are the pixel values ​​of four integer coordinate points within the neighborhood of the original image; The nearest integer coordinates are used; this interpolation operation ensures that the scaled image slices retain the texture details and edge features of the foreign object.

[0094] Material feature coding:

[0095] The classifier uses a lightweight feature extraction network. For encoding, the lightweight feature extraction network is specially configured with a channel attention mechanism to enhance the response weights to material specular highlights, reflectivity, and surface texture features, thereby distinguishing between "metal / glass" (high reflectivity, high hardness) and "wood / plastic" (diffuse reflection, medium hardness) or "fabric" (low-frequency texture, low hardness). The network ends with a global average pooling layer to process the 3D feature map output from the final convolutional layer. Compressed into one-dimensional feature vectors Let feature map The size is ,in If is the number of channels, then the th Output characteristic values ​​of each channel The calculation is as follows:

[0096] ;

[0097] in, Indicates the first Each feature map in spatial coordinates The activation value at the point of activation, global average pooling operation establishes a direct link between features and categories, and reduces the number of model parameters.

[0098] Attribute probability calculation:

[0099] Feature vector The input is fed into a fully connected layer, and finally output through a Softmax function layer, showing the probability distribution of each material risk category. Let the output logical value vector of the fully connected layer be... ,in The total number of predefined material categories (in this embodiment) ), then the first Posterior probability of material The calculation formula is: ;

[0100] The classifier outputs the index of the class with the highest probability. This serves as the final material determination result for the foreign object.

[0101] This classifier uses a weighted cross-entropy loss function for parameter optimization during the training phase to address the problem of imbalanced sample numbers for different materials. The loss function... Defined as:

[0102] ;

[0103] in, For the one-hot encoding of the true label, if the sample belongs to the first... Class Otherwise, it is 0; For the first The preset weight coefficients for class samples are set to be larger for high-risk classes with scarce samples. The value is used to improve the model's sensitivity to that category.

[0104] See attached document Figure 2To ensure the robustness of the foreign object detection and classification model under complex in-vehicle lighting conditions and to accurately distinguish the physical properties of different materials, this embodiment constructs a dedicated dataset containing multi-dimensional annotation information and designs a composite loss function with geometric constraints for iterative optimization of model parameters.

[0105] Dataset Construction and Augmentation: The process of constructing a dataset is divided into three stages: data collection, fine annotation, and data augmentation.

[0106] Data collection phase: By deploying data collection devices in the driver's cab of different vehicle types (sedans, SUVs, commercial vehicles), image samples covering all-weather lighting conditions (direct sunlight on sunny days, diffused sunlight on cloudy days, low illumination in tunnels, and interior lighting at night) are obtained. The collection targets mainly cover four categories of items: soft items (such as plush toys and tissue boxes), sturdy items (such as wooden prayer beads and resin ornaments), hard items (such as metal phone holders and diamond-encrusted steering wheel covers), and hard and fragile items (such as glass perfume holders and crystal handicrafts).

[0107] During the data annotation stage, rectangular bounding boxes are used. Mark the location of the target in the image and assign a category label at the same time. and material attribute tags

[0108] In the data augmentation stage, in order to expand the diversity of the samples and prevent the model from overfitting, Mosaic data augmentation technology is introduced before training. Mosaic data augmentation technology randomly selects four training images, performs random scaling, random cropping and random arrangement respectively, and finally stitches them together to synthesize a new training sample image. Mosaic data augmentation technology increases the number of targets contained in a single input image, enabling the model to learn context-independent features of targets at different scales during training.

[0109] Composite loss function design: A composite loss function is used during the backpropagation process of model training. To measure the difference between model predictions and true labeled values, the composite loss function is used. It consists of three weighted parts: location loss Confidence loss and classification loss .

[0110] ;

[0111] in, , and These are the balance coefficients for localization, confidence, and classification tasks, respectively.

[0112] Location loss To address the issues of traditional IoU loss functions failing to handle the non-intersection of predicted and ground truth bounding boxes and their slow convergence, this embodiment employs the CloU loss function. CloU adds a center point distance penalty and an aspect ratio consistency penalty to IoU, and its calculation formula is as follows:

[0113] ;

[0114] in, The intersection-union ratio (IUU) between the predicted bounding box and the ground truth bounding box is calculated as follows: ; and : These represent the center point coordinate vectors of the predicted bounding box and the ground truth bounding box, respectively; To calculate the Euclidean distance between two points; The diagonal length of the smallest bounding rectangle that can simultaneously contain both the predicted bounding box and the ground truth bounding box; The aspect ratio influencing factor is defined as follows: ; The parameter used to measure aspect ratio consistency is calculated using the following formula:

[0115] ;

[0116] in, The width and height of the actual bounding box. To predict the width and height of the bounding box, the CloU loss function makes the model not only pursue the maximization of the overlap area when regressing the bounding box, but also force the center point of the predicted box to move closer to the ground truth center and maintain the consistency of the aspect ratio.

[0117] Confidence loss With classification loss Both the confidence loss and classification loss are calculated using binary cross-entropy loss. For the confidence loss... It is responsible for measuring whether a target exists within a grid cell, as shown in the following formula:

[0118] ;

[0119] in, The total number of samples; The logical confidence value output by the network; For the Sigmoid function; For the true label (if the grid cell is responsible for predicting the target, then) ,otherwise ).

[0120] For classification loss Its calculation is performed only on positive samples (i.e., grids containing the target) and is used to measure the accuracy of class prediction.

[0121] Parameter optimization process: Model parameters are updated using stochastic gradient descent (SGD) or the AdamW optimizer. The training process incorporates a cosine annealing learning rate scheduling strategy, ensuring the learning rate... With the number of iterations The learning rate decays as a cosine function to achieve rapid convergence in the early stages of training and to finely search for the global optimum in the later stages. The formula for the learning rate change is:

[0122] ;

[0123] in, and These are the initial learning rate and the minimum learning rate, respectively. This represents the current iteration round number; The total number of iterations represents the total number of iterations. Through this optimization process, the weight matrix and bias terms in the neural network are continuously updated until the loss function is reached. Converging to below the preset threshold.

[0124] See attached document Figure 2 The pose estimation submodule is configured to execute a multi-person 2D pose estimation algorithm based on a bottom-up mechanism. This algorithm does not rely on pre-defined human detection boxes, but directly predicts the positions of all visible human body parts in parallel on the full-resolution feature map of the input image. Then, it assembles these parts into a complete human skeleton through an association algorithm. Based on the bottom-up mechanism, it is particularly suitable for the cockpit environment and can effectively handle situations where the front passenger is partially obscured or in a non-standard sitting posture (such as curled up or lying on their side).

[0125] The pose estimation submodule employs a two-branch, multi-stage convolutional neural network (CNN) architecture. The network input is a preprocessed image tensor. The network's front end is a shared feature extraction backbone network, consisting of the first 10 VGG-19 or MobileNet convolutional layers, used to generate basic feature maps. Following the backbone network, the network splits into two parallel processing branches, each containing... A series of cascaded prediction stages.

[0126] Branch 1: This branch is configured to predict a set of keypoint confidence maps. Each confidence plot Corresponding to a specific anatomical key point in the human body (e.g., right ankle, left knee), the value of each pixel in the image represents the confidence probability that the pixel belongs to a specific body part.

[0127] Branch 2: This branch is configured to predict a set of part affinity fields. PAFs are a set of two-dimensional vector fields used to encode the position and orientation information of limbs. Each affinity field... A type of limb connection corresponding to the human body (For example, the connection between "right knee and right ankle").

[0128] During the training phase, in order to supervise the learning of branch one, it is necessary to generate a true confidence map based on the coordinates of the actually labeled keypoints. For the first The first of the individuals The key points have the following true coordinates: The corresponding true confidence plot In image location The value at that point is defined using a two-dimensional Gaussian distribution function: ;

[0129] in, The standard deviation parameter is used to control the spread range of the Gaussian peak. It is the Euclidean norm.

[0130] For images containing multiple people, the overall true confidence map Take the maximum value of the response values ​​of all individuals at that location: ;

[0131] During the inference phase, the network outputs a prediction confidence graph. After non-maximum suppression processing, the system searches for local maxima within a local window. If the response value of a pixel is greater than a preset threshold and is a local maximum, then the pixel is identified as a candidate key point location.

[0132] To correctly connect adjacent keypoints belonging to the same person (e.g., distinguishing the driver's left hand from the passenger's right hand), branch two uses vector fields to model the limbs, connecting keypoints... and limbs Its corresponding real part affinity field Any point within the limb area At that point, it is defined as a unit vector. The direction of this unit vector is from point to :

[0133] ;

[0134] If point If it is located within the rectangular area of ​​the limb, then ;otherwise .

[0135] During the assembly phase, for the two candidate key points and The system calculates the line integral between them. To assess the probability that these two points belong to the same limb:

[0136] ;

[0137] in, It is the linear interpolation position between two candidate points:

[0138] ;

[0139] By calculating the integral value between all candidate keypoint pairs We construct a weighted bipartite graph and use the Hungarian algorithm or a greedy matching strategy to perform maximum weight matching, thereby assembling isolated keypoints into a complete human skeleton set.

[0140] The training objective of the network is to minimize the L2 distance between the predicted and ground truth maps. The total loss function is... Two branches in each stage The weighted sum of the losses is composed of:

[0141] ;

[0142] in, and The first Confidence plot and affinity field of stage prediction; The total number of key points; This represents the total number of limb connections. For binary masks, when the image position The value is 0 if there is a lack of labeled information, and 1 otherwise, to avoid interference from unlabeled areas during training.

[0143] Through the above process, the pose estimation submodule ultimately outputs the set of skeletal keypoint coordinates for the co-driver. These coordinates are transmitted to the subsequent logic judgment module to identify specific dangerous postures.

[0144] See attached document Figure 2 The rule-based judgment logic for dangerous behavior outputs the set of skeletal key point coordinates of the co-driver passenger in the attitude estimation submodule. Then, the system executes a set of rule-based judgment logic based on geometric constraints to identify predefined dangerous driving behaviors. These logical rules do not rely on deep learning models, but are implemented by directly performing mathematical operations and threshold comparisons on the coordinates of key points, ensuring the real-time nature and interpretability of the judgment process.

[0145] The judgment of a passenger placing their legs on the center console is based on whether the passenger's knees and ankles have entered a predefined danger zone above the dashboard.

[0146] First, mark a convex polygon representing the danger zone of the center console in the vehicle coordinate system. The polygon consists of a set of vertex coordinates. These coordinates are defined and mapped to the image coordinate system.

[0147] For from Extracted key points of the passenger's right ankle and key points of the right knee The test function executes when the point is inside the polygon. This function uses the ray casting method to determine points. Is it located in a polygon? If the point is inside the interior, the function returns 1; otherwise, it returns 0.

[0148] The behavior of placing both legs on the center console Determined by the following logical expression:

[0149] ;

[0150] in, and The key coordinates are the left ankle and left knee, respectively. This expression indicates that a dangerous behavior is determined to have occurred when the key points of the passenger's left or right knee and ankle are simultaneously located within the danger zone polygon of the center console.

[0151] The determination of a passenger not wearing a seatbelt employs a strategy that combines onboard CAN bus signals with visual-assisted verification. The primary basis for judgment is the seatbelt buckle status signal obtained from the vehicle data interface unit 30. The signal is a binary value, where 0 represents "not engaged" and 1 represents "engaged". When the signal is unavailable or a fault is reported, the system initiates visual-assisted verification. This verification process is performed within a predefined region of interest along the seatbelt path. The operation is carried out internally. This area roughly covers a strip of space from the passenger's shoulder to the side of their abdomen, and is first performed on... The image within is processed using the Canny edge detection algorithm to generate an edge map. Subsequently, The Hough transform is performed to detect lines. The Hough transform maps lines in image space to parameter space. The point in the middle.

[0152] ;

[0153] in, The perpendicular distance from the origin to the line is... Let be the angle between the vertical line and the x-axis.

[0154] By searching for accumulator peaks in the parameter space, the main straight line segments in the image can be identified. If the angle of inclination of the most prominent detected straight line segment is... Within the preset normal seat belt tilt angle range Inside, the visual judgment marker position Set to 1, indicating the behavior of not wearing a seatbelt. Determined by the following logic:

[0155] ;

[0156] Where ¬ represents the logical NOT operation; This is a flag indicating whether the CAN signal is available. The expression indicates that a dangerous behavior is determined to have occurred when the CAN signal explicitly indicates that the seat belt is not fastened, or when the CAN signal is unavailable and the seat belt is not visually detected.

[0157] The judgment of a passenger's prolonged forward leaning behavior combines spatial position thresholds and time cumulative counting. First, a vertical safety threshold line is defined in the image coordinate system. Its position is determined by the x-axis. This line represents the safety boundary that a passenger's front body should not cross in a normal sitting posture. In each frame of the image, the system acquires the passenger's shoulder key points. and And calculate the coordinates of its midpoint. :

[0158] ;

[0159] Compare the x-coordinates of the midpoint With safety threshold line The system maintains a forward-tilt state duration counter. In processing the first At frame time, the following update logic is executed:

[0160] ;

[0161] Prolonged forward-leaning behavior The duration is determined by comparing a counter with a preset time threshold, where the set duration threshold is... (Unit: seconds), image acquisition frame rate The state is then determined as follows:

[0162] ;

[0163] This expression indicates that only when the passenger's midpoint continuously crosses the safety threshold line for a period of time reaches or exceeds [a certain threshold] will the safety threshold be met. It took several seconds to finally determine that the dangerous behavior had occurred, thus avoiding false alarms caused by normal actions such as passengers briefly retrieving items.

[0164] See attached document Figure 3 Before calculating the comprehensive risk score, the dynamic risk assessment and decision-making module 23 first assigns a quantified basic risk weight to each identified risk source. This weight represents the inherent static risk of the risk source without considering vehicle dynamics (i.e., when the vehicle speed is zero). These weight values ​​are pre-set and stored in the module's non-volatile memory.

[0165] For dangerous driving behaviors, a weight set is defined. In one specific embodiment, these weights are set as follows: weight for the passenger placing their legs on the center console. Its value is set to 100; the weight of passenger not wearing a seatbelt. Its value was set to 70; weighting of passengers' prolonged forward-leaning behavior. Its value is set to 30.

[0166] The above assignments are based on the differences in the severity of occupant injury caused by different behaviors in a frontal collision. Legs-crossing poses the highest risk of disability or death in a collision, and therefore carries the highest weight. Not wearing a seatbelt significantly increases the risk of occupants being ejected or experiencing secondary impacts with hard points inside the vehicle. Leaning forward causes the occupant's head to enter the initial high-pressure danger zone of airbag deployment, but its risk level is lower than the former two. Therefore, the weighting follows the following relationship: .

[0167] For hazardous foreign objects placed in the airbag deployment area, their basic risk weight is determined by their material properties, and a set of material risk weights is defined. In one specific embodiment, these weights are set as follows: weight for hard and fragile types (such as glass and crystal). Its value is set to 60; weight for hard materials (such as metals and ceramics). Its value is set to 40; weight for sturdy classes (such as wood, hard plastic). Its value is set to 20; the weight of soft items (such as plush toys, tissues) is... Its value is set to 5.

[0168] The weighting is based on the potential damage of an item as it transforms into a projectile when the airbag deploys at high speed. Hard and fragile items not only have high kinetic energy but may also produce sharp fragments that can cause puncture or lacerations, thus receiving the highest weight. Hard items have high kinetic energy and impact force, followed by solid items. Soft items, due to their light weight, easy deformation, and low kinetic energy conversion efficiency, have the lowest damage potential. Therefore, the weighting follows this relationship:

[0169] ;

[0170] In addition, a risk gain weight is set for foreign objects in the steering wheel area. The risk gain weight for the steering wheel area is set because the space around the steering wheel is smaller, the airbag deployment pressure is higher, and the projectiles are aimed directly at the driver's head and face, posing a greater direct threat. In a specific embodiment, this gain weight is set as: Steering Wheel Area Risk Gain Weight Its value is set to 20, and all the above basic risk weights are included. The parameters are pre-stored in the parameter table of the dynamic risk assessment and decision-making module 23 as the original input values ​​for subsequent calculation of the basic risk score.

[0171] See attached document Figure 3 After obtaining the basic risk score, the dynamic risk assessment and decision-making module 23 further introduces a speed adjustment factor. The speed adjustment factor is a quantitative multiplier used to scale the base risk score. The introduction of the speed adjustment factor is based on the fact that the vehicle's kinetic energy is proportional to the square of its speed. Higher driving speed means that the vehicle has higher collision energy in the event of a collision. This energy level directly affects the airbag control unit's initiation decision threshold and deployment intensity, thereby amplifying the severity of secondary injuries caused by foreign objects or improper postures.

[0172] To model this physical relationship, the velocity adjustment factor f It is defined as a piecewise function that takes the continuously acquired vehicle speed input value in real time. (Unit: km / h), based on its relationship with two preset speed thresholds and The relationship is mapped to one of three discrete risk adjustment coefficient values, and the mathematical expression of this piecewise function is as follows:

[0173] ;

[0174] in, The velocity adjustment factor is a dimensionless value. The vehicle speed is obtained in real time from the vehicle data interface unit 30; The threshold for dividing the low-speed and medium-speed ranges; This is the threshold for dividing the medium-speed and high-speed ranges; In the low-speed range Risk adjustment coefficient for internal applications; In the medium speed range Risk adjustment coefficient for internal applications; In the high-speed range Risk adjustment coefficient for internal applications.

[0175] In one specific embodiment of the present invention, the above parameters are assigned the following values:

[0176] Speed ​​threshold The speed threshold is set at 60 km / h, which corresponds to the typical speed limit for urban roads; Set at 90 km / h, this value corresponds to the lower speed limit for expressways or some highways; risk adjustment coefficient. Set to 1.0. This value indicates that the risk is not amplified under low-speed urban driving conditions; risk adjustment coefficient. The risk adjustment factor is set at 1.5. The value is set to 2.0. These coefficients follow a monotonically increasing relationship, that is: .

[0177] This relationship ensures that the amplification effect of the risk score increases progressively as the vehicle moves from the low-speed zone to the medium-speed zone and then to the high-speed zone. The dynamic risk assessment and decision-making module 23 acquires the real-time vehicle speed in each calculation cycle. Substituting into the piecewise function above, the velocity adjustment factor corresponding to the current moment is calculated. The value is then used to multiply the base risk score to generate a dynamically adjusted risk measure.

[0178] The dynamic risk assessment and decision-making module 23 is configured to perform a deterministic calculation process to output a final comprehensive risk score. This process first calculates a base risk score that aggregates all identified static risk sources, and then dynamically weights it using a velocity adjustment factor.

[0179] First, define a basic risk score. It is calculated as the sum of the basic risk weights of all dangerous behaviors and hazardous objects detected at the current moment. The calculation formula is as follows:

[0180] ;

[0181] in, The basic risk score is a dimensionless numerical value. , , These are status flags for three behaviors: legs crossed, not wearing a seatbelt, and leaning forward for an extended period of time. When the corresponding behavior is detected, its value is 1; when it is not detected, its value is 0. , , These are the basic risk weights corresponding to the three behaviors mentioned above; The total number of hazardous foreign objects detected within the airbag deployment area; For all The risk of each detected foreign object is summed; : for the first The material risk weight of each foreign object is determined based on the foreign object classification result. Selected from; For the first A foreign object location marker is used. When the foreign object is located in the steering wheel area, its value is 1; when it is located in other areas (such as the passenger-side center console), its value is 0. The risk gain weight for the steering wheel area.

[0182] After calculating the basic risk score Then, the module combines it with the speed adjustment factor. Multiply the results to obtain the final comprehensive risk score. The calculation formula is as follows: ;

[0183] in: A dimensionless numerical value is used to calculate the overall risk score, which serves as the direct basis for tiered early warning. To be based on real-time vehicle speed The calculated speed adjustment factor.

[0184] The following example illustrates the calculation process:

[0185] Scenario Description: The vehicle is traveling at 100 km / h on a highway. The front passenger has their legs resting on the dashboard, which also has a glass perfume holder on it. Calculation Steps:

[0186] Identify the activated risk sources and their parameters:

[0187] Behavior detection: Legs crossed behavior was detected, therefore the status flag is set. No other dangerous behaviors were detected, therefore .

[0188] Foreign object detection: A foreign object was detected in the center console area, therefore The foreign object is a glass perfume holder, which belongs to the "hard and fragile" category; therefore, its material risk weight is... The foreign object is not located in the steering wheel area, therefore its location marker is not there. .

[0189] Vehicle speed data: Real-time vehicle speed obtained from vehicle data interface unit 30 km / h.

[0190] Calculate the basic risk score Substitute the above parameters into the basic risk score calculation formula:

[0191] ;

[0192] The calculation result is: .

[0193] Therefore, basic risk classification The calculated value is 160.

[0194] Determine the speed adjustment factor According to the piecewise function definition of the speed adjustment factor, due to the current vehicle speed... km / h greater than the high-speed threshold km / h, therefore the risk adjustment coefficient for high-speed sections is applied. :Right now Therefore, the speed adjustment factor The value is 2.0.

[0195] Calculate the overall risk score :Will and Substitute the calculation result into the final score formula:

[0196] ;

[0197] Finally, the comprehensive risk score is calculated from the vehicle data interface unit 30. The overall risk score is 320, and will be passed on to the subsequent tiered strategy module.

[0198] See attached document Figure 3 The comprehensive risk score is calculated in the dynamic risk assessment and decision-making module 23. Subsequently, the dynamic risk assessment and decision-making module 23 performs a threshold comparison operation, mapping the continuous numerical risk score to discrete hazard level signals. The danger level signal is used to trigger the specific response strategies preset in the subsequent early warning and intervention control module 24, which correspond to each level.

[0199] This mapping process will... This is achieved by comparing the values ​​with a predefined, strictly monotonically increasing set of numerical thresholds. In one specific embodiment of the invention, three key thresholds are defined: ,and These thresholds divide the entire risk score range into five non-overlapping sub-ranges, each corresponding to one of the five risk levels: (Very low risk) (Low risk) (Medium risk) (High risk) and (Extremely high risk), danger level signal The generation follows the following piecewise function definition:

[0200] ;

[0201] in, The output is a discrete hazard level signal; The overall risk score is the input. The threshold for classifying low and medium risk; The threshold for classifying medium and high risk; This is the threshold for distinguishing between high risk and extremely high risk.

[0202] In one specific embodiment, these thresholds are set to the following values:

[0203] , , ;

[0204] These thresholds are set based on the following criteria:

[0205] threshold This threshold is used to distinguish between isolated, low-potential risk sources and risk sources with clear potential for harm. For example, a soft object moving at low speed... Or a sturdy object traveling at low speed ( All are below this threshold and are classified as While passengers lean forward during low-speed travel ( If it just reaches that threshold, it is also classified as... However, when the vehicle speed enters the medium speed range ( If 1.5 = 45, then it will exceed the threshold and enter... .

[0206] threshold This threshold is used to isolate a single source of risk with serious consequences, such as not wearing a seatbelt while driving at low speeds. ) in However, after entering the medium speed range ( If it exceeds the threshold, it will be upgraded to... .

[0207] threshold This threshold is used to identify the highest level of risk scenarios, which are usually caused by a combination of high-risk behavior and high-speed driving, or by the superposition of multiple risk sources, such as the act of crossing one's legs while driving at a medium speed. When the threshold is reached, it is classified as When the vehicle speed enters the high-speed range ( If it exceeds the threshold, it will enter... In the example scenario in the previous section, the calculated comprehensive risk score... Since 320 > 150, this scenario is classified as the highest level of danger. .

[0208] By setting this quantitative threshold, the system can transform complex, multi-factor-affected risk situations into structured, directly usable level signals for decision control, thereby providing deterministic input for subsequent execution of precise and appropriate early warning and intervention actions.

[0209] See attached document Figure 3 The early warning and intervention control module 24 receives discrete hazard level signals. It implements a predefined, hierarchical, and progressive response strategy. The response strategy aims to ensure that the intensity of the warning matches the risk level, providing gentle, non-disruptive alerts at low risk and taking decisive, strong warnings and interventions at high risk.

[0210] (Very low risk) ):

[0211] Triggering condition: The system does not detect any dangerous behavior or dangerous foreign objects.

[0212] Execution strategy: The system is in a silent monitoring state. The warning and intervention control module 24 does not generate any output commands, the human-machine interface (HMI) remains unchanged, and the vehicle control system does not take any intervention actions.

[0213] (Low risk, ):

[0214] Triggering conditions: Usually triggered by a single, low-inherent-risk event while driving at low speeds, such as placing a soft object in the passenger airbag area.

[0215] Execution strategy:

[0216] Visual warning: In the real-time video feed on the central control display, when a foreign object is detected, the system renders a semi-transparent, fixed, non-flickering yellow border. At the same time, a static, small yellow triangle warning icon is lit up in the status bar area of ​​the instrument panel or central control screen.

[0217] Audio warning: None. No sound is produced at this level to avoid unnecessary interference with the driver.

[0218] Objective: To provide contextual awareness so that occupants are aware of potential risks, but without triggering an emergency alarm.

[0219] (Medium risk) ):

[0220] Triggering conditions: Usually triggered by a single risk source with clear harm potential, or by multiple low-risk sources combined, such as a passenger not wearing a seatbelt while driving at low speed, or a hard object being placed in the passenger airbag area.

[0221] Implementation Strategy: Visual Warning: The visual warning intensity is upgraded. The system-rendered bounding box turns orange and begins to flash at a low frequency (e.g., 1Hz). The warning icon in the status bar also flashes synchronously. At the same time, a short text prompt, such as "Please clear items in front of the passenger seat" or "Please fasten your seatbelt," is superimposed next to the video screen. Audio Warning: The system triggers a short, gentle alert sound at a moderate volume to attract the occupant's attention. Objective: To clearly warn the occupant that the risk has increased and that attention and corrective measures are required.

[0222] (High risk) ):

[0223] Triggering conditions: Usually triggered by behaviors with high inherent risk (such as crossing one's legs) or medium-risk sources while traveling at high speed.

[0224] Execution Strategy: Visual Warning: Visual warnings reach high intensity. The system-rendered bounding box turns a striking red and flashes rapidly at a high frequency (e.g., 3Hz). A prominent warning window pops up on the central control screen, covering some non-critical information and displaying strong warnings such as "High risk! Please correct dangerous behavior immediately!"; Audio Warning: Audio warnings are upgraded to continuous, rapid beeps, accompanied by clear, loud synthesized voice announcements. The content is directly related to the risk source, such as: "Detected that the passenger's legs are dangling, posing a serious danger!"

[0225] Vehicle Intervention: Upon initial activation of active vehicle intervention, the warning and intervention control module 24 sends a CAN bus message to the body control module via the vehicle data interface unit 30, instructing the vehicle to automatically activate its hazard lights to alert other road users to the abnormal condition of the vehicle. The structure is as follows:

[0226] ;

[0227] in: For message frame ID; Data length code; For data fields.

[0228] Objective: To issue strong, unmistakable warnings to compel occupants to immediately cease dangerous behavior and to alert them to potential risks via external vehicle signals.

[0229] (Extremely high risk) )

[0230] Triggering conditions: Usually caused by the superposition of multiple high-risk sources, or by a single high-risk source when traveling at high speed, for example, when a passenger puts their legs on the center console while traveling at high speed.

[0231] Implementation strategy: Inherit and enhance visual and audio alerts. All warning measures included increased frequency of voice announcements and a change from a buzzer to a more penetrating siren; vehicle intervention: in Building upon this foundation, a higher level of intervention can be added. The system can be configured to send a command to the seatbelt control module to trigger the seatbelt pretensioner to perform one or more brief point-controlled tightening actions. This action conveys an extremely dangerous signal to the passenger in the most direct way by generating a noticeable tactile pulling force, forcing them to return to a normal sitting posture. This intervention is also achieved by sending a CAN message with specific content.

[0232] HMI Takeover: The system can be configured to temporarily take over part of the central control screen display area, displaying a large warning symbol with a red background in the entire screen until the risk level is reduced.

[0233] Objective: To activate all available warning and intervention measures to minimize the risk of impending, foreseeable serious harm.

[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visually detecting foreign objects and dangerous driving behaviors in a cab, comprising: The method comprises the following steps: ​ S1, acquiring image data in the cab and real-time vehicle speed of the vehicle, and based on the acquired image data, identifying dangerous foreign matter located in a preset airbag deployment area and determining the material property of the dangerous foreign matter, and identifying dangerous driving behavior of a passenger and determining the type of the dangerous driving behavior; S2, calculating a basic risk score according to the determined material property of the dangerous foreign matter and the determined type of the dangerous driving behavior, determining a speed adjustment factor according to the acquired real-time vehicle speed, and multiplying the calculated basic risk score and the determined speed adjustment factor to obtain a comprehensive risk score; S3, mapping the obtained comprehensive risk score to a preset danger level signal, and executing warning and intervention measures corresponding to the danger level signal according to the mapped danger level signal.

2. The method of claim 1, wherein the method further comprises: In the S1 step, the step of identifying the dangerous foreign matter and determining the material property of the dangerous foreign matter comprises: processing the acquired image data through a single-stage target detection network architecture to output a candidate bounding box containing the position of the foreign matter; processing the image region in the output candidate bounding box through an independent cascaded risk property classifier to determine the material property of the dangerous foreign matter.

3. The method of claim 1, wherein the method further comprises: In the S1 step, the step of identifying the dangerous driving behavior of the passenger and determining the type of the dangerous driving behavior comprises: processing the acquired image data through a bottom-up pose estimation submodule to obtain a set of skeletal key point coordinates of the passenger; based on the obtained set of skeletal key point coordinates, executing a set of geometric constraint-based regularization judgment logic to identify a predefined dangerous driving behavior.

4. The method of claim 1, wherein the method further comprises: In the S2 step, the step of calculating the basic risk score comprises: setting a basic risk weight for each of the predefined types of dangerous driving behavior and the material properties of dangerous foreign matter; adding up the basic risk weights corresponding to all the identified dangerous driving behaviors and dangerous foreign matters at the current time to obtain the basic risk score.

5. The method of claim 4, wherein the method further comprises: In the S2 step, the step of calculating the basic risk score further comprises: when the identified dangerous foreign matter is located in the steering wheel area, additionally adding a preset regional risk gain weight to the dangerous foreign matter when adding up the basic risk weights.

6. The method of claim 1, wherein the method further comprises: In the S2 step, the step of determining the speed adjustment factor comprises: comparing the acquired real-time vehicle speed with at least one preset speed threshold to obtain a comparison result; according to the comparison result, selecting one of a plurality of preset risk adjustment coefficients that monotonically increase with the speed interval as the speed adjustment factor.

7. The method of claim 1, wherein the method further comprises: In the S3 step, the step of mapping the obtained comprehensive risk score to a preset danger level signal comprises: comparing the obtained comprehensive risk score with a set of preset strictly monotonically increasing numerical thresholds to obtain the numerical interval in which the comprehensive risk score is located; according to the numerical interval in which the comprehensive risk score is located, determining a corresponding discrete danger level as the danger level signal.

8. The method of claim 3, wherein the method further comprises: The step of the set of geometric constraint-based regularization judgment logic comprises: The coordinates of the passenger knee key points and ankle key points in the obtained set of skeleton key point coordinates are compared with a predefined dangerous area in terms of spatial position to determine whether a dangerous driving behavior exists due to the passenger's lower limbs entering the predefined dangerous area.

9. The method of claim 1, wherein the method further comprises: In the S3 step, the pre-warning and intervention measures include: when the mapped danger level signal is low risk or medium risk, performing visual pre-warning, which includes rendering a high-light bounding box on the real-time video image of the center control display screen for the identified dangerous foreign matter.

10. The method of claim 1, wherein the method further comprises: In the S3 step, the pre-warning and intervention measures include: when the mapped danger level signal is high risk or extremely high risk, performing vehicle intervention, which includes sending an instruction to the vehicle body control module to turn on the double flash warning light, and sending an instruction to the safety belt control module to trigger the safety belt pretensioner to perform point control type tightening.

Images