A driving object state monitoring method, device, equipment and storage medium

Abstract

The application discloses a driving object state monitoring method and device, equipment and storage medium, and relates to the technical field of computer vision, comprising: collecting a driver video stream in a vehicle-mounted scene, extracting video frame features by relying on an improved deep convolutional neural network embedded with a deformable attention mechanism, accurately positioning the driver's face, eyes and hand targets in a complex environment with low light, camera angle deviation and limited imaging area; then, the eye state is classified and detected, and when the closed-eye feature is recognized, the human posture estimation module is synchronously triggered to construct the skeleton key point atlas of the driver's head and limbs; finally, multi-dimensional semantic analysis is carried out in combination with the geometric correlation of the head pitch angle and the hand spatial position, so as to logically determine and distinguish the distraction state of the driver lowering his head to play the mobile phone and the real fatigue state. The application significantly improves the robustness and recognition accuracy of the driver state monitoring.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and storage medium for monitoring the state of a driving object. Background Technology

[0002] With the increase in car ownership, fatigued driving has become one of the main causes of traffic accidents. Existing driver monitoring systems (DMS) mainly rely on computer vision technology to determine fatigue status by detecting the degree of eye closure of the driver, such as the PERCLOS (Percentage of Eyelid Closure over the Pupil over Time) index.

[0003] However, existing technical solutions face severe challenges in practical automotive applications: First, poor environmental adaptability: The automotive environment is extremely complex. For example, in tunnels or at night, lighting conditions change drastically, and vehicle vibrations or driver seat adjustments often cause camera angle shifts, tilts, or partial obstructions. Traditional convolutional neural networks (CNNs) have fixed geometric receptive fields, making it difficult to adapt to such non-rigid image deformations, resulting in decreased ability to extract key features such as eyes and hands, and limited detection accuracy. Second, high false alarm rate for "false fatigue": When drivers are "looking down at their phones," their gaze naturally falls downwards, and their eyelids close or partially close. Detection systems relying solely on facial features are prone to misjudging this "distraction" as "drowsiness." Although these two states have similar facial features, their dangerous nature and warning strategies are completely different. The former requires prompting to concentrate, while the latter requires forced rest; existing single-modal detection methods struggle to effectively distinguish between them.

[0004] Therefore, how to develop a monitoring solution that can adapt to complex in-vehicle environments and accurately distinguish between fatigue and distraction states through multi-dimensional feature collaborative analysis is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for monitoring the state of a driving object, enabling a monitoring scheme that can adapt to complex in-vehicle environments and accurately distinguish between fatigue and distraction states through multi-dimensional feature collaborative analysis. The specific scheme is as follows: Firstly, this application provides a method for monitoring the state of a driving object, including: A behavior dataset of driving objects in various vehicle scenarios is constructed, and a deep convolutional neural network containing a deformable attention mechanism is constructed. The deep convolutional neural network is trained using the behavior dataset and a composite loss function to obtain an object detection model. During the driving process of the target driver, the video stream of the target driver captured by the vehicle-mounted camera is acquired, and the video frames in the video stream are input into the target detection model to obtain the detection box information of the target driver's eye state, mouth state, head region and hand region output by the target detection model; If the eye state is closed and the duration of the closed state is greater than a preset duration threshold, a corresponding risk signal is generated to activate a preset human posture estimation component, and the first coordinate information of the preset skeleton key points of the target driving object in the video frame is determined based on the preset human posture estimation component and the detection box information. Based on the detection box information and the first coordinate information, the head posture vector and the second coordinate information of the wrist region of the target driving object are determined, and the pitch angle of the line connecting the nose tip region and the neck region of the target driving object relative to the preset vertical coordinate axis is determined based on the head posture vector. Based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest regarding the chest operation, it is determined whether the target driving object in the video frame exhibits a preset abnormal state; the preset abnormal state includes an abnormal state of distraction and fatigue.

[0006] Optionally, the construction of the deep convolutional neural network including a deformable attention mechanism includes: Define the tensor operation process for the basic building blocks of deep convolutional neural networks, define the splitting logic for cross-stage local networks, and define the operation logic for deformable attention mechanisms. Based on the tensor operation process, the splitting logic, the operation logic, the basic building blocks, and the cross-stage local network, a deep convolutional neural network incorporating the deformable attention mechanism is constructed.

[0007] Optionally, the definition of the tensor operation process regarding the basic building blocks of the deep convolutional neural network includes: Determine the input feature tensors for the basic building blocks of a deep convolutional neural network; The input feature tensor, the weights of the convolution kernel in the deep convolutional neural network, the output channel of the convolution kernel, and the bias term of the output channel are used to generate the feature value of the input feature tensor at the target position of the output channel, so as to determine the corresponding tensor convolution process; Using the feature values, the mean and variance of batch data with respect to the input feature tensor, and preset learnable parameters, a normalized tensor is generated to determine the corresponding tensor normalization process. The normalized tensor and the preset activation function are used to generate an output feature tensor to determine the corresponding tensor activation process; Based on the tensor convolution process, the tensor normalization process, and the tensor activation process, the tensor operation process for the basic building blocks of the deep convolutional neural network is defined.

[0008] Optionally, the definition of the cross-stage local network splitting logic includes: Determine the input features for cross-stage local networks; In the channel of the cross-stage local network, the input features are subjected to a preset segmentation operation to obtain segmented features, so as to determine the corresponding channel segmentation process; The process of processing features using the preset residual components of the cross-stage local network is determined as the main path, and the process of prohibiting feature processing is determined as the cross path. The segmented features are input into the main path and the cross path respectively to obtain the processed features, so as to determine the corresponding dual-path processing flow. The processed features are spliced ​​together to obtain spliced ​​features, and the transition layer of the cross-stage local network is used to integrate the spliced ​​features to obtain integrated features, so as to determine the corresponding feature fusion process; The traffic splitting logic for cross-stage local networks is defined based on the channel segmentation process, the dual-path processing process, and the feature fusion process.

[0009] Optionally, the operational logic defining the deformable attention mechanism includes: Determine the query feature vector related to the deformable attention mechanism; The fully connected layer of the deep convolutional neural network predicts the offset and attention weight of the target attention head and target sampling point corresponding to the query feature vector. The fourth coordinate information of the target sampling point is determined using the third coordinate information of the preset reference point and the offset, and the feature value corresponding to the fourth coordinate information is determined using bilinear interpolation. The attention output vector is determined based on the preset projection matrix, the preset number of sampling points, the offset, the attention weight, and the feature value, so as to define the operational logic of the deformable attention mechanism.

[0010] Optionally, after obtaining the detection bounding box information of the target driving object's eye state, mouth state, head region, and hand region output by the target detection model, the method further includes: A time series of the eye condition is constructed, and based on the time series, an eye confidence level and a condition category are determined for the eye condition; the condition category includes an open-eye state and a rhinitis state. The state variables of the eye state are determined based on the eye confidence, the state category, and the preset confidence threshold, and the proportion of closed eyes within the preset time window is determined based on the preset time window and the state variables. Based on the stated proportion of closed eyes, it is determined whether the eye state is closed and whether the duration of the closed state is greater than a preset duration threshold.

[0011] Optionally, determining the head pose vector and the second coordinate information of the wrist region of the target driving object based on the detection box information and the first coordinate information includes: Based on the detection frame information and the first coordinate information, the head posture vector and torso posture vector of the target driving object are determined, and the fifth coordinate information of the shoulder region and the second coordinate information of the wrist region of the target driving object are determined from the detection frame information and the first coordinate information.

[0012] Optionally, determining the pitch angle of the line connecting the nose tip region and neck region of the target driving object relative to a preset vertical coordinate axis based on the head posture vector includes: Based on the head posture vector and the torso posture vector, the cosine value of the angle between the line connecting the nose tip region and the neck region of the target driving object and the preset vertical coordinate axis is determined, and the corresponding pitch angle is determined based on the cosine value of the angle.

[0013] Optionally, determining whether the target driving object in the video frame exhibits a preset abnormal state based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest regarding the chest operation includes: Based on the fifth coordinate information, the shoulder width of the target driving object is determined, and the shoulder width, preset neck key points, and preset area scaling factor are used to construct a region of interest about chest operation; Based on the first coordinate information, the second coordinate information, the pitch angle, and the region of interest, the state of the target driving object in the video frame is determined, and a corresponding determination result is obtained; If the determination result indicates that the pitch angle is greater than a preset angle threshold and the wrist area is within the region of interest, then the target driving object is determined to be in a distracted state in the video frame. If the determination result indicates that the pitch angle is not greater than the preset angle threshold or the wrist area is not in the region of interest, and the eye state is the closed state, then the target driving object is determined to be in a fatigued state in the video frame.

[0014] Secondly, this application provides a state monitoring device for a driving object, comprising: The network training module is used to construct a behavior dataset of driving objects in various vehicle scenarios, and to construct a deep convolutional neural network containing a deformable attention mechanism. The deep convolutional neural network is trained using the behavior dataset and a composite loss function to obtain a target detection model. The information output module is used to acquire the video stream of the target driving object captured by the vehicle-mounted camera during the driving process, and input the video frames in the video stream to the target detection model to obtain the detection box information of the target driving object's eye state, mouth state, head region and hand region output by the target detection model; The information extraction module is used to generate a corresponding risk signal if the eye state is closed and the duration of the closed state is greater than a preset duration threshold, so as to activate a preset human posture estimation component using the risk signal, and determine the first coordinate information of the preset skeleton key points of the target driving object in the video frame based on the preset human posture estimation component and the detection box information. The pitch angle determination module is used to determine the head posture vector and the second coordinate information of the wrist region of the target driving object based on the detection box information and the first coordinate information, and to determine the pitch angle of the line connecting the nose tip region and the neck region of the target driving object relative to the preset vertical coordinate axis based on the head posture vector. The status monitoring module is used to determine whether the target driving object in the video frame is in a preset abnormal state based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest related to the chest operation; the preset abnormal state is an abnormal state that includes a distracted state and a fatigued state.

[0015] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned method for monitoring the state of a driving object.

[0016] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for monitoring the state of a driving object.

[0017] In this application, a behavior dataset of driving objects in various vehicle scenarios is constructed, and a deep convolutional neural network including a deformable attention mechanism is constructed. The deep convolutional neural network is trained using the behavior dataset and a composite loss function to obtain a target detection model. During the driving process of the target driving object, a video stream of the target driving object is acquired by an in-vehicle camera device, and the video frames in the video stream are input into the target detection model to obtain the detection box information of the target driving object's eye state, mouth state, head region, and hand region output by the target detection model. If the eye state is closed and the duration of the closed state is greater than a preset duration threshold, a corresponding risk signal is generated to activate a preset... A human pose estimation component is used to determine the first coordinate information of a preset skeleton key point of the target driving object in the video frame based on the preset human pose estimation component and the detection box information; the head pose vector and the second coordinate information of the wrist region of the target driving object are determined based on the detection box information and the first coordinate information, and the pitch angle of the line connecting the nose tip region and the neck region of the target driving object relative to a preset vertical coordinate axis is determined based on the head pose vector; the pitch angle, the first coordinate information, the second coordinate information, and the region of interest about the chest operation are used to determine whether the target driving object in the video frame has a preset abnormal state; the preset abnormal state is an abnormal state that includes a distracted state and a fatigued state. As can be seen from the above, this application first acquires real-time video streams of the driver in an in-vehicle scenario, and then extracts video frame features using an improved deep convolutional neural network with embedded deformable attention mechanisms. This allows for precise localization of the driver's face, eyes, and hands in complex environments with low light, camera viewpoint shifts, and limited imaging areas. Subsequently, eye status is classified and detected. When closed-eye features are identified, a human pose estimation module is simultaneously triggered to construct a skeletal keypoint map of the driver's head and limbs. Finally, multi-dimensional semantic analysis is conducted based on the geometric correlation between head pitch angle and hand spatial position to logically determine and distinguish between the driver's distracted state of looking down at a mobile phone and a true state of fatigue. In this way, this application effectively solves the technical pain points of single-vision detection technology in complex in-vehicle scenarios, such as weak anti-interference ability and high false alarm rate, significantly improving the robustness and accuracy of driver status monitoring. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart of a method for monitoring the state of a driving object disclosed in this application; Figure 2 This is a flowchart of a specific method for monitoring the state of a driving object disclosed in this application; Figure 3 This is a schematic diagram of the head posture and region of interest of a driving object as disclosed in this application; Figure 4 This is a schematic diagram of the structure of a driving object status monitoring device disclosed in this application; Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Currently, existing technologies for monitoring driver status face significant challenges in practical in-vehicle applications: First, poor environmental adaptability: In-vehicle environments are extremely complex, such as in tunnels or at night, where lighting conditions change drastically. Vehicle vibrations or driver seat adjustments often cause camera angle shifts, tilts, or partial obstructions. Traditional convolutional neural networks (CNNs) have fixed geometric receptive fields, making them ill-suited to such non-rigid image deformations. This leads to a decreased ability to extract key features such as eyes and hands, limiting detection accuracy. Second, a high false alarm rate for "false fatigue": When drivers are looking down at their phones, their gaze naturally falls downwards, and their eyelids close or partially close. Detection systems relying solely on facial features easily misjudge this "distraction" as "drowsiness." Although these two states have similar facial features, their dangerous nature and warning strategies are completely different. The former requires prompting to concentrate, while the latter requires forced rest; existing single-modal detection methods struggle to effectively distinguish between them.

[0022] See Figure 1 As shown, this embodiment of the invention discloses a method for monitoring the state of a driving object, including: Step S11: Construct a behavior dataset of driving objects in various vehicle scenarios, and construct a deep convolutional neural network containing a deformable attention mechanism. Use the behavior dataset and a composite loss function to train the deep convolutional neural network to obtain an object detection model.

[0023] In this embodiment, a behavior dataset for model training needs to be constructed first. This dataset should cover the behavior data of driving objects in various in-vehicle scenarios. After constructing the behavior dataset, a deep convolutional neural network incorporating a deformable attention mechanism needs to be built. Specifically, the tensor operation process for the basic building blocks of the deep convolutional neural network is defined, the branching logic for cross-stage local networks is defined, and the operation logic for the deformable attention mechanism is defined. Based on the tensor operation process, the branching logic, the operation logic, the basic building blocks, and the cross-stage local networks, a deep convolutional neural network incorporating the deformable attention mechanism is constructed.

[0024] In one specific implementation, when defining the tensor operation process for the basic building blocks of a deep convolutional neural network, the following steps are taken: First, an input feature tensor for the basic building blocks of the deep convolutional neural network is determined. Then, a feature value of the input feature tensor at the target position of the output channel is generated using the input feature tensor, the weight values ​​of the convolution kernel in the deep convolutional neural network, the output channel of the convolution kernel, and the bias term of the output channel, to determine the corresponding tensor convolution process. Next, a normalized tensor is generated using the feature value, the mean and variance of batch data related to the input feature tensor, and a preset learnable parameter, to determine the corresponding tensor normalization process. Finally, an output feature tensor is generated using the normalized tensor and a preset activation function, to determine the corresponding tensor activation process. Based on the tensor convolution process, the tensor normalization process, and the tensor activation process, the tensor operation process for the basic building blocks of the deep convolutional neural network is defined.

[0025] In one specific implementation, when defining the splitting logic of a cross-stage local network, the input features of the cross-stage local network are determined; in the channels of the cross-stage local network, the input features are subjected to a preset segmentation operation to obtain segmented features, thereby determining the corresponding channel segmentation process; the process of processing the features using the preset residual components of the cross-stage local network is determined as the main path, and the process of prohibiting the processing of features is determined as the traversal path; the segmented features are respectively input to the main path and the traversal path to obtain processed features, thereby determining the corresponding dual-path processing process; the processed features are concatenated to obtain concatenated features, and the concatenated features are integrated using the transition layer of the cross-stage local network to obtain integrated features, thereby determining the corresponding feature fusion process; the splitting logic of the cross-stage local network is defined based on the channel segmentation process, the dual-path processing process, and the feature fusion process.

[0026] In one specific implementation, when defining the operational logic of the deformable attention mechanism, a query feature vector for the deformable attention mechanism is determined; the offset and attention weight of the target attention head and target sampling point corresponding to the query feature vector are predicted through the fully connected layer of the deep convolutional neural network; the fourth coordinate information of the target sampling point is determined using the third coordinate information of a preset reference point and the offset, and the feature value corresponding to the fourth coordinate information is determined using bilinear interpolation; the attention output vector is determined based on a preset projection matrix, a preset number of sampling points, the offset, the attention weight, and the feature value, so as to define the operational logic of the deformable attention mechanism.

[0027] After constructing the deep convolutional neural network, it was trained using the pre-built behavior dataset. During training, a composite loss function was used to optimize the network parameters. Upon completion of training, a target detection model was obtained, which can be used for subsequent real-time monitoring of the driving object's state.

[0028] Step S12: During the driving process of the target driver, acquire the video stream of the target driver captured by the vehicle-mounted camera, and input the video frames in the video stream into the target detection model to obtain the detection box information of the target driver's eye state, mouth state, head region and hand region output by the target detection model.

[0029] In this embodiment, the vehicle-mounted camera is used to capture images of the driver's face and upper body. During the driver's journey, a video stream of the driver is acquired by the vehicle-mounted camera. After acquiring the video stream, frame extraction is performed to convert the continuous video stream into a discrete sequence of video frames. Each frame is then input into a pre-trained target detection model.

[0030] When processing video frames, the deformable attention mechanism in the object detection model adaptively focuses on key regions in the image relevant to the driver's state monitoring. Through forward propagation calculations, the model ultimately outputs bounding box information for the driver's eye state, mouth state, head region, and hand region. The eye state includes information on the degree of eye opening and closing; the head region bounding box is used to locate the position and extent of the driver's head in the image; and the hand region bounding box is used to locate the position and extent of the driver's hands in the image. This bounding box information can be represented in coordinate form, for example, using the pixel coordinates of the top-left and bottom-right corners of the bounding box to characterize its position and size. By acquiring this multi-dimensional state information, foundational data support can be provided for subsequent driving behavior analysis and state monitoring.

[0031] Step S13: If the eye state is closed and the duration of the closed state is greater than a preset duration threshold, a corresponding risk signal is generated to activate a preset human posture estimation component, and the first coordinate information of the preset skeleton key points of the target driving object in the video frame is determined based on the preset human posture estimation component and the detection box information.

[0032] In this embodiment, after obtaining the eye state output by the target detection model, it is necessary to continuously monitor the eye state to determine whether the risk signal generation conditions are met.

[0033] To accurately determine whether the eye is in a closed state and whether the duration of the closed state exceeds a preset threshold, a time series of the eye state can be constructed. Based on the time series, an eye confidence level and state category for the eye state can be determined. The state category includes open-eye state and rhinitis state. Based on the eye confidence level, the state category, and the preset confidence threshold, a state variable for the eye state can be determined. Based on a preset time window and the state variable, the proportion of closed eyes within the preset time window can be determined. Based on the proportion of closed eyes, it can be determined whether the eye state is closed and whether the duration of the closed state exceeds a preset duration threshold.

[0034] If the eye state is determined to be closed and the duration of the closed state is greater than a preset duration threshold, a corresponding risk signal is generated. The risk signal is then used to activate a preset human posture estimation component. Based on the preset human posture estimation component and the detection box information, the first coordinate information of the preset skeleton key points of the target driving object in the video frame is determined.

[0035] Step S14: Based on the detection box information and the first coordinate information, determine the head posture vector and the second coordinate information of the wrist region of the target driving object, and based on the head posture vector, determine the pitch angle of the line connecting the nose tip region and the neck region of the target driving object relative to the preset vertical coordinate axis.

[0036] In this embodiment, after obtaining the first coordinate information of the preset skeletal key points of the target driving object, it is necessary to further determine the head posture-related parameters. Specifically, based on the detection box information and the first coordinate information, the head posture vector and torso posture vector of the target driving object are determined, and the fifth coordinate information of the shoulder region and the second coordinate information of the wrist region of the target driving object are determined from the detection box information and the first coordinate information.

[0037] After determining the head posture vector, the pitch angle of the target driver is further determined based on the head posture vector. Specifically, the cosine value of the angle between the line connecting the nose tip region and the neck region of the target driver and a preset vertical coordinate axis is determined based on the head posture vector and the torso posture vector, and the corresponding pitch angle is determined based on the cosine value. By combining the head posture vector and the torso posture vector for pitch angle calculation, interference caused by vehicle bumps or changes in vehicle body posture can be eliminated, improving the accuracy of head posture monitoring.

[0038] Step S15: Based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest regarding the chest operation, determine whether the target driving object in the video frame exhibits a preset abnormal state; the preset abnormal state is an abnormal state that includes a distracted state and a fatigued state.

[0039] In this embodiment, after obtaining the pitch angle and the second coordinate information of the wrist region, it is necessary to comprehensively judge the state of the driving object by combining it with a preset region of interest. In order to better determine whether the target driving object is in a preset abnormal state, the shoulder width of the target driving object is determined based on the fifth coordinate information, and a region of interest about the chest operation is constructed using the shoulder width, preset neck key points, and preset region scaling factor.

[0040] After constructing the region of interest, the state of the target driving object in the video frame is judged based on the first coordinate information, the second coordinate information, the pitch angle, and the region of interest, and the corresponding judgment result is obtained.

[0041] In one specific implementation, if the judgment result indicates that the pitch angle is greater than a preset angle threshold and the wrist area is within the region of interest, then the target driver's state in the video frame is determined to be a distracted state. In another specific implementation, if the judgment result indicates that the pitch angle is not greater than the preset angle threshold or the wrist area is not within the region of interest, and the eye state is closed, then the target driver's state in the video frame is determined to be a fatigued state. Through the above multi-dimensional comprehensive judgment mechanism, distracted behavior and fatigued behavior of the driver can be effectively distinguished, providing a more accurate state basis for subsequent early warning intervention.

[0042] As can be seen from the above, this application first acquires real-time video streams of the driver in an in-vehicle scenario, and then extracts video frame features using an improved deep convolutional neural network with embedded deformable attention mechanisms. This allows for precise localization of the driver's face, eyes, and hands in complex environments with low light, camera viewpoint shifts, and limited imaging areas. Subsequently, eye status is classified and detected. When closed-eye features are identified, a human pose estimation module is simultaneously triggered to construct a skeletal keypoint map of the driver's head and limbs. Finally, multi-dimensional semantic analysis is conducted based on the geometric correlation between head pitch angle and hand spatial position to logically determine and distinguish between the driver's distracted state of looking down at a mobile phone and a true state of fatigue. In this way, this application effectively solves the technical pain points of single-vision detection technology in complex in-vehicle scenarios, such as weak anti-interference ability and high false alarm rate, significantly improving the robustness and accuracy of driver status monitoring.

[0043] Reference Figure 2 and Figure 3 As shown below, the technical solutions of the embodiments of this application will be described in detail.

[0044] First, an improved object detection model for complex in-vehicle environments is constructed and trained. This step may include: constructing a driver behavior dataset containing various complex in-vehicle scenarios, including low-light environments, highly blurred videos, large-angle tilted camera views, and partial occlusion of the driver's face; the dataset labels include target bounding box categories for mouth, eyes (for determining open / closed eyes), and hands (for determining whether a mobile phone is being held); building an improved deep convolutional neural network architecture, embedding a deformable attention module (i.e., Deformable AttentionModule) in the backbone feature extraction network, with the Deformable AttentionModule embedded at the deep output (high-level features) of the backbone network; this module is used to automatically learn sampling offsets based on the spatial correlation of the input feature maps to generate feature maps that adapt to camera view distortion and target pose changes; using the dataset, the improved deep convolutional neural network is trained end-to-end, and a composite loss function including localization loss and classification loss is used for iterative optimization to obtain the trained improved object detection model.

[0045] The backbone feature extraction network adopts the CSPNet (Cross Stage Partial Network) architecture and is constructed from basic convolutional units (CBS, Convolution-BatchNorm-Silu). Each CBS unit contains convolutional layers, BN (Batch Normalization) layers, and the SiLU activation function. Compared to the traditional ReLU, it has a non-zero gradient in the negative region, which can better preserve the dark feature information in low-light images; the operation process of the basic convolutional unit is defined by the tensor operation definition of the basic convolutional unit, the split logic definition of the cross-stage partial network (i.e., CSP), and the operation logic definition of the multi-head deformable attention mechanism.

[0046] For tensor operations that define a basic convolutional unit, let the input feature tensor be... , C in The number of channels in the input feature map. H The height of the input feature map, W The width of the input feature map. Given the set of real numbers; the output feature tensor is The computational process includes: convolution operation, defining the convolution kernel weights as follows: , C out The number of channels in the output feature map. K Let be the spatial size of the convolution kernel; with a bias of b, the formula for calculating the feature value of the c-th output channel at position (i, j) is as follows: ; The computation process also includes batch normalization, which normalizes the convolution output. Perform channel-level normalization, setting the mean of the current batch. and variance The learnable parameters are Then the normalized output The calculation formula is as follows: ; In the formula, It is a very small constant used to prevent the denominator from being zero and to ensure numerical stability.

[0047] The computational process also includes: SiLU activation, for Perform an element-wise non-linear mapping to generate the final output. The calculation formula is as follows: ; In the formula, For element-wise multiplication, Sigmoid is the activation function.

[0048] For the branching logic of the cross-stage local network, let the input characteristics be... The operational logic of the CSP structure includes: channel splitting. ,in, , C The number of channels for the input feature. These are the two sub-feature tensors obtained after channel segmentation. Dual-path processing, main branch: That is, it is processed through N residual modules; the path spans (i.e., the cross branch): Directly retain the original features; feature fusion (i.e., concatenation): That is, after concatenating the channels, gradient information is integrated through a transition layer.

[0049] The specific computational logic of the Multi-Head Deformable Attention mechanism includes bilinear interpolation sampling of non-grid points, and the sampling logic is as follows: Linear projection generation of sampling parameters. Let the query feature vector be... The coordinates of the reference point are Predicting the th through a fully connected layer (i.e., Linear) The first attention head, the first Offset of each sampling point and attention weight Furthermore, the offset is a real number, such as... This means that the sampling points can "move" to positions between pixels, close to the edge of the target. The formulas for calculating the offset and attention weights are as follows: ; In the formula, For the fully connected layer used to predict offsets, This is a fully connected layer used to predict attention weights.

[0050] After obtaining the offset, the target coordinates are defined based on feature sampling using bilinear interpolation. ,in The coordinates are real numbers. Furthermore, due to the sampling coordinates... Since the values ​​are not integers, the computer cannot directly read the pixel values. This embodiment uses the bilinear interpolation algorithm. Assume the sampling point coordinates are... Its pixel value is determined by the surrounding four integer coordinate points. The pixel values ​​are weighted together to obtain the result. In other words, let... U=u , V=vIf the coordinates are integers rounded down, then the eigenvalues ​​of the non-integer positions are... From its four neighboring pixels The weighted calculation yields: ; The specific expansion calculation formula is as follows: ; In the formula, Representing coordinates eigenvalues ​​at that location Representing coordinates The eigenvalue at that location.

[0051] The method for determining the aforementioned feature values ​​gives the network "geometric adaptability." When the camera is tilted, causing the driver's face to appear skewed in the image, the sampling points will automatically tilt accordingly, always focusing on key information such as the corners of the eyes and mouth, thus solving the environmental adaptability problem.

[0052] After obtaining the aforementioned feature values, the weighted aggregation of multi-head features is calculated based on the feature values ​​obtained from the interpolation, resulting in the final multi-head attention output. : ; In the formula, and The projection matrix is ​​learnable. For the number of heads, This represents the number of sampling points.

[0053] Secondly, the video frames to be tested are acquired and preliminary feature perception is performed. This step may include: real-time acquisition of the video stream to be tested from the vehicle-mounted camera, and inputting the current frame into the trained improved target detection model; the model outputs the confidence score of the driver's eye state category (which may include closed / open) in the current frame, as well as the coordinates of the target detection boxes for the head and hands.

[0054] Then, multi-level state analysis is performed based on a conditional triggering mechanism. This step may include: performing temporal statistics on eye state, and generating a "suspected fatigue trigger signal" when the eyes are detected to be in a closed state for a duration exceeding a preset threshold; in response to the trigger signal, activating the human posture estimation module to extract skeletal key points of the driver's upper body in the current frame, and obtaining a two-dimensional key point coordinate sequence including the nose tip, head, shoulders, etc.

[0055] For the conditional triggering mechanism, the PERCLOS (Percentage of Eyelid Closure over Time) discrimination logic based on a time sliding window is used as the condition for triggering pose estimation. Its operation logic is as follows: Constructing a time series of eye conditions by setting a time sliding window ,Include Frame image; for the first Frame images, based on the output eye confidence score. and categories Define eye state variables The formula for calculating the eye state variable is as follows: ; In the formula, This is the confidence threshold.

[0056] Calculate the percentage of frames with eyes closed (PERCLOS) within a statistical sliding window (e.g., 30 frames per second). : ; The formula for the percentage of eyes closed can also be: The conditional triggering logic defines the fatigue trigger threshold. For example, 40%, that is, only when When the system detects a potential driver malfunction, it activates the attitude estimation module. The formula for determining whether to activate the module is as follows: ; In the above formula, the subsequent human pose estimation steps are executed only when the output is TRUE, in order to reduce the average computing power consumption of the system.

[0057] Specifically, the triggered human pose estimation module is used to construct a two-dimensional vector space for the driver's upper body, and is defined as follows: Define the set of key points for the driver's upper body These correspond to the tip of the nose, neck, left shoulder, right shoulder, left wrist, and right wrist, respectively; each key point Includes coordinate information and visibility confidence .

[0058] Skeleton vectorization is based on keypoint coordinates, constructing a feature vector for pose analysis: the head pose vector. TorsoVector (base vector): (Define a vertically downward unit vector as the reference); Hand position coordinates: .

[0059] Next, geometric features are fused for multi-dimensional semantic determination. This step may include: constructing a head pose vector and calculating the pitch angle of the line connecting the nose tip keypoint and the neck keypoint relative to the vertical coordinate axis; constructing hand spatial region determination logic and calculating whether the wrist keypoint falls within the preset "Region of Interest (ROI)"; executing state differentiation logic: if the calculated pitch angle is greater than the preset head-down threshold and the wrist keypoint is located within the chest ROI, it is determined that the downward gaze is caused by the head-down operation of the device, and the output evaluation result is "distracted driving (which may include playing on a mobile phone)"; if the calculated pitch angle does not exceed the head-down threshold, or the wrist keypoint is not located within the chest ROI, but a continuous eye-closing feature is detected in the step of performing time-series statistics on eye state, the output evaluation result is "fatigue driving".

[0060] Among them, the head pitch angle (Pitch Angle Calculation) is calculated using the vector dot product formula to calculate the head posture vector. With vertical reference vector The angle between : ; ; like Less than the preset angle threshold If the vector deflects downwards, it is determined to be in a "head-down state".

[0061] The region of interest (ROI) for hand manipulation is defined based on key neck points. and shoulder width Dynamically construct the "chest operating area". : ; In the formula, This is the region scaling factor.

[0062] Among them, the fusion decision tree is used to determine the driver's state. The final logical deduction is performed, and the decision logic for condition A (i.e., the distraction judgment) is as follows: ; The above logic explains that although the driver's eyes were closed, their head was lowered and their hands were in the chest area, indicating that the downward gaze was caused by using a mobile phone.

[0063] The decision logic for condition B (i.e., fatigue determination) is as follows: ; The above logic explains that if the posture characteristics of playing on a mobile phone are not met, but the rate of closing the eyes is still very high, it is judged as true fatigue.

[0064] Finally, the final driver health status assessment results are output and corresponding early warning strategies are implemented.

[0065] Furthermore, it should be noted that the technical solution of this application can run on an in-vehicle embedded computing platform, such as the NVIDIA Jetson series, Horizon Robotics series, or an in-vehicle platform. The system hardware includes a near-infrared (IR) camera installed on the A-pillar or rearview mirror, an in-vehicle computing unit, and an alarm device.

[0066] It is evident that this application introduces a deformable attention mechanism, which improves the model's detection accuracy under conditions of camera vibration, displacement, or low light. By incorporating pose estimation and geometric logic, it successfully solves the problem of the traditional PERCLOS algorithm misclassifying "looking down at a phone" as "fatigue and closed eyes." Employing a conditional triggering mechanism, pose evaluation is initiated only when an anomaly is suspected. Compared to running a multimodal model continuously, this reduces the system's average resource consumption by 40%, making it suitable for deployment on vehicle-mounted devices.

[0067] Accordingly, see Figure 4 As shown, this application embodiment provides a driving object status monitoring device, including: The network training module 11 is used to construct a behavior dataset of driving objects in various vehicle scenarios, and to construct a deep convolutional neural network containing a deformable attention mechanism. The deep convolutional neural network is trained using the behavior dataset and a composite loss function to obtain a target detection model. The information output module 12 is used to acquire the video stream of the target driving object collected by the vehicle-mounted camera device during the driving process of the target driving object, and input the video frames in the video stream to the target detection model to obtain the detection box information of the target driving object's eye state, mouth state, head area and hand area output by the target detection model; The information extraction module 13 is used to generate a corresponding risk signal if the eye state is closed and the duration of the closed state is greater than a preset duration threshold, so as to activate a preset human posture estimation component using the risk signal, and determine the first coordinate information of the preset skeleton key points of the target driving object in the video frame based on the preset human posture estimation component and the detection box information. The pitch angle determination module 14 is used to determine the head posture vector and the second coordinate information of the wrist region of the target driving object based on the detection box information and the first coordinate information, and to determine the pitch angle of the line connecting the nose tip region and the neck region of the target driving object relative to the preset vertical coordinate axis based on the head posture vector. The status monitoring module 15 is used to determine whether the target driving object in the video frame has a preset abnormal state based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest related to the chest operation; the preset abnormal state is an abnormal state that includes a distracted state and a fatigued state.

[0068] In some specific embodiments, the network training module 11 specifically includes: The logic definition unit is used to define the tensor operation process of the basic building blocks of deep convolutional neural networks, define the splitting logic of cross-stage local networks, and define the operation logic of deformable attention mechanisms. A network construction unit is used to construct a deep convolutional neural network containing the deformable attention mechanism based on the tensor operation process, the splitting logic, the operation logic, the basic building blocks, and the cross-stage local network.

[0069] In some specific embodiments, the logical definition unit specifically includes: Tensor determination subunits are used to determine the input feature tensors for the basic building blocks of deep convolutional neural networks; The first process determines the sub-unit, which is used to generate the feature value of the input feature tensor at the target position of the output channel using the input feature tensor, the weight value of the convolution kernel in the deep convolutional neural network, the output channel of the convolution kernel, and the bias term of the output channel, so as to determine the corresponding tensor convolution process; The second process determination subunit is used to generate a normalized tensor using the feature values, the mean and variance of batch data about the input feature tensor, and preset learnable parameters, so as to determine the corresponding tensor normalization process. The third process determines the sub-unit, which is used to generate an output feature tensor using the normalized tensor and the preset activation function, so as to determine the corresponding tensor activation process. The operation definition subunit is used to define the tensor operation process for the basic building blocks of the deep convolutional neural network based on the tensor convolution process, the tensor normalization process, and the tensor activation process.

[0070] In some specific embodiments, the logical definition unit specifically includes: The feature determination subunit is used to determine the input features of the cross-stage local network; The fourth process determination subunit is used to perform a preset segmentation operation on the input features in the channel of the cross-stage local network to obtain segmented features, so as to determine the corresponding channel segmentation process; The path determination subunit is used to determine the process of processing features using the preset residual components of the cross-stage local network as the main path, and to determine the process of prohibiting feature processing as the cross path. The fifth process determination subunit is used to input the segmented features into the main path and the cross path respectively to obtain the processed features, so as to determine the corresponding dual-path processing flow; The sixth process determines the sub-unit, which is used to splice the processed features to obtain spliced ​​features, and integrate the spliced ​​features using the transition layer of the cross-stage local network to obtain integrated features, so as to determine the corresponding feature fusion process. The first logic definition subunit is used to define the traffic splitting logic of the cross-stage local network based on the channel segmentation process, the dual-path processing process, and the feature fusion process.

[0071] In some specific embodiments, the logical definition unit specifically includes: Vector-determining subunits are used to determine the query feature vectors related to the deformable attention mechanism; The information determination subunit is used to predict the offset and attention weight of the target attention head and target sampling point corresponding to the query feature vector through the fully connected layer of the deep convolutional neural network; The feature value determination subunit is used to determine the fourth coordinate information of the target sampling point using the third coordinate information of the preset reference point and the offset, and to determine the feature value corresponding to the fourth coordinate information using bilinear interpolation. The second logic definition subunit is used to determine the attention output vector based on the preset projection matrix, the preset number of sampling points, the offset, the attention weight, and the feature value, so as to define the operation logic of the deformable attention mechanism.

[0072] In some specific embodiments, the information output module 12 further includes: A state information determination unit is used to construct a time series of the eye state, and determine the eye confidence level and state category based on the time series; the state category includes an open-eye state and a rhinitis state. The proportion determination unit is used to determine the state variable of the eye state based on the eye confidence, the state category and the preset confidence threshold, and to determine the proportion of closed eyes within the preset time window based on the preset time window and the state variable. The information judgment unit is used to determine whether the eye state is closed and whether the duration of the closed state is greater than a preset duration threshold based on the closed eye ratio.

[0073] In some specific embodiments, the pitch angle determination module 14 specifically includes: The coordinate information determination unit is used to determine the head posture vector and torso posture vector of the target driving object based on the detection frame information and the first coordinate information, and to determine the fifth coordinate information of the shoulder region and the second coordinate information of the wrist region of the target driving object from the detection frame information and the first coordinate information.

[0074] In some specific embodiments, the pitch angle determination module 14 specifically includes: The pitch angle determination unit is used to determine the cosine value of the angle between the line connecting the nose tip region and the neck region of the target driving object and a preset vertical coordinate axis based on the head posture vector and the torso posture vector, and to determine the corresponding pitch angle based on the cosine value.

[0075] In some specific embodiments, the status monitoring module 15 specifically includes: The region determination unit is used to determine the shoulder width of the target driving object based on the fifth coordinate information, and to construct a region of interest about the chest operation using the shoulder width, preset neck key points and preset region scaling factor; A state determination unit is used to determine the state of the target driving object in the video frame based on the first coordinate information, the second coordinate information, the pitch angle, and the region of interest, and to obtain a corresponding determination result. The first state determination unit is used to determine that the state of the target driving object in the video frame is a distracted state if the determination result shows that the pitch angle is greater than a preset angle threshold and the wrist area is in the region of interest. The second state determination unit is used to determine that the target driving object is in a fatigued state in the video frame if the determination result shows that the pitch angle is not greater than the preset angle threshold or the wrist area is not in the region of interest and the eye state is the closed state.

[0076] Furthermore, embodiments of this application also disclose an electronic device, Figure 5This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the driving object state monitoring method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0077] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0078] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0079] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the driving object status monitoring method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0080] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for monitoring the state of a driving object. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0082] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0084] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0085] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for monitoring the state of a driving object, characterized in that, include: A behavior dataset of driving objects in various vehicle scenarios is constructed, and a deep convolutional neural network containing a deformable attention mechanism is constructed. The deep convolutional neural network is trained using the behavior dataset and a composite loss function to obtain an object detection model. During the driving process of the target driver, the video stream of the target driver captured by the vehicle-mounted camera is acquired, and the video frames in the video stream are input into the target detection model to obtain the detection box information of the target driver's eye state, mouth state, head region and hand region output by the target detection model; If the eye state is closed and the duration of the closed state is greater than a preset duration threshold, a corresponding risk signal is generated to activate a preset human posture estimation component, and the first coordinate information of the preset skeleton key points of the target driving object in the video frame is determined based on the preset human posture estimation component and the detection box information. Based on the detection box information and the first coordinate information, the head posture vector and the second coordinate information of the wrist region of the target driving object are determined, and the pitch angle of the line connecting the nose tip region and the neck region of the target driving object relative to the preset vertical coordinate axis is determined based on the head posture vector. Based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest regarding the chest operation, determine whether the target driving object in the video frame exhibits a preset abnormal state; The preset abnormal state is an abnormal state that includes both distraction state and fatigue state.

2. The method for monitoring the state of a driving object according to claim 1, characterized in that, The construction of the deep convolutional neural network including a deformable attention mechanism includes: Define the tensor operation process for the basic building blocks of deep convolutional neural networks, define the splitting logic for cross-stage local networks, and define the operation logic for deformable attention mechanisms. Based on the tensor operation process, the splitting logic, the operation logic, the basic building blocks, and the cross-stage local network, a deep convolutional neural network incorporating the deformable attention mechanism is constructed.

3. The method for monitoring the state of a driving object according to claim 2, characterized in that, The definition of the tensor operation process for the basic building blocks of a deep convolutional neural network includes: Determine the input feature tensors for the basic building blocks of a deep convolutional neural network; The input feature tensor, the weights of the convolution kernel in the deep convolutional neural network, the output channel of the convolution kernel, and the bias term of the output channel are used to generate the feature value of the input feature tensor at the target position of the output channel, so as to determine the corresponding tensor convolution process; Using the feature values, the mean and variance of batch data with respect to the input feature tensor, and preset learnable parameters, a normalized tensor is generated to determine the corresponding tensor normalization process. The normalized tensor and the preset activation function are used to generate an output feature tensor to determine the corresponding tensor activation process; Based on the tensor convolution process, the tensor normalization process, and the tensor activation process, the tensor operation process for the basic building blocks of the deep convolutional neural network is defined.

4. The method for monitoring the state of a driving object according to claim 2, characterized in that, The defined traffic splitting logic for cross-stage local networks includes: Determine the input features for cross-stage local networks; In the channel of the cross-stage local network, the input features are subjected to a preset segmentation operation to obtain segmented features, so as to determine the corresponding channel segmentation process; The process of processing features using the preset residual components of the cross-stage local network is determined as the main path, and the process of prohibiting feature processing is determined as the cross path. The segmented features are input into the main path and the cross path respectively to obtain the processed features, so as to determine the corresponding dual-path processing flow. The processed features are spliced ​​together to obtain spliced ​​features, and the transition layer of the cross-stage local network is used to integrate the spliced ​​features to obtain integrated features, so as to determine the corresponding feature fusion process; The traffic splitting logic for cross-stage local networks is defined based on the channel segmentation process, the dual-path processing process, and the feature fusion process.

5. The method for monitoring the state of a driving object according to claim 2, characterized in that, The operational logic defining the deformable attention mechanism includes: Determine the query feature vector related to the deformable attention mechanism; The fully connected layer of the deep convolutional neural network predicts the offset and attention weight of the target attention head and target sampling point corresponding to the query feature vector. The fourth coordinate information of the target sampling point is determined using the third coordinate information of the preset reference point and the offset, and the feature value corresponding to the fourth coordinate information is determined using bilinear interpolation. The attention output vector is determined based on the preset projection matrix, the preset number of sampling points, the offset, the attention weight, and the feature value, so as to define the operational logic of the deformable attention mechanism.

6. The method for monitoring the state of a driving object according to claim 1, characterized in that, After obtaining the detection bounding box information of the target driving object's eye state, mouth state, head region, and hand region output by the target detection model, the method further includes: A time series of the eye condition is constructed, and based on the time series, an eye confidence level and a condition category are determined for the eye condition; the condition category includes an open-eye state and a rhinitis state. The state variables of the eye state are determined based on the eye confidence, the state category, and the preset confidence threshold, and the proportion of closed eyes within the preset time window is determined based on the preset time window and the state variables. Based on the stated proportion of closed eyes, it is determined whether the eye state is closed and whether the duration of the closed state is greater than a preset duration threshold.

7. The method for monitoring the state of a driving object according to any one of claims 1 to 6, characterized in that, The step of determining the head pose vector and wrist region coordinates of the target driving object based on the detection box information and the first coordinate information includes: Based on the detection frame information and the first coordinate information, the head posture vector and torso posture vector of the target driving object are determined, and the fifth coordinate information of the shoulder region and the second coordinate information of the wrist region of the target driving object are determined from the detection frame information and the first coordinate information.

8. The method for monitoring the state of a driving object according to claim 7, characterized in that, The step of determining the pitch angle of the line connecting the nose tip region and neck region of the target driver relative to a preset vertical coordinate axis based on the head posture vector includes: Based on the head posture vector and the torso posture vector, the cosine value of the angle between the line connecting the nose tip region and the neck region of the target driving object and the preset vertical coordinate axis is determined, and the corresponding pitch angle is determined based on the cosine value of the angle.

9. The method for monitoring the state of a driving object according to claim 7, characterized in that, The step of determining whether the target driving object in the video frame exhibits a preset abnormal state based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest regarding the chest operation includes: Based on the fifth coordinate information, the shoulder width of the target driving object is determined, and the shoulder width, preset neck key points, and preset area scaling factor are used to construct a region of interest about chest operation; Based on the first coordinate information, the second coordinate information, the pitch angle, and the region of interest, the state of the target driving object in the video frame is determined, and a corresponding determination result is obtained; If the determination result indicates that the pitch angle is greater than a preset angle threshold and the wrist area is within the region of interest, then the target driving object is determined to be in a distracted state in the video frame. If the determination result indicates that the pitch angle is not greater than the preset angle threshold or the wrist area is not in the region of interest, and the eye state is the closed state, then the target driving object is determined to be in a fatigued state in the video frame.

10. A device for monitoring the state of a driving object, characterized in that, include: The network training module is used to construct a behavior dataset of driving objects in various vehicle scenarios, and to construct a deep convolutional neural network containing a deformable attention mechanism. The deep convolutional neural network is trained using the behavior dataset and a composite loss function to obtain a target detection model. The information output module is used to acquire the video stream of the target driving object captured by the vehicle-mounted camera during the driving process, and input the video frames in the video stream to the target detection model to obtain the detection box information of the target driving object's eye state, mouth state, head region and hand region output by the target detection model; The information extraction module is used to generate a corresponding risk signal if the eye state is closed and the duration of the closed state is greater than a preset duration threshold, so as to activate a preset human posture estimation component using the risk signal, and determine the first coordinate information of the preset skeleton key points of the target driving object in the video frame based on the preset human posture estimation component and the detection box information. The pitch angle determination module is used to determine the head posture vector and the second coordinate information of the wrist region of the target driving object based on the detection box information and the first coordinate information, and to determine the pitch angle of the line connecting the nose tip region and the neck region of the target driving object relative to the preset vertical coordinate axis based on the head posture vector. The status monitoring module is used to determine whether the target driving object in the video frame has a preset abnormal state based on the pitch angle, the first coordinate information, the second coordinate information, and the region of interest about the chest operation; The preset abnormal state is an abnormal state that includes both distraction state and fatigue state.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the state monitoring method for a driving object as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the state monitoring method for a driving object as described in any one of claims 1 to 9.

Images