A vehicle double-driving wheel change monitoring method and system based on multi-feature fusion

Abstract

The application discloses a kind of based on multi-feature fusion vehicle double driver rotation monitoring method and system, comprising: acquisition cabin multi-source data, pre-process synchronization, generate three kinds of feature sequences;Visual sequence is sent into improved RTMDet, and extracts cabin and control related features;Three-element structure diagram is constructed, and topological reasoning identifies control sequence and rotation interval;Potential energy is calculated in combination with multiple features, and rotation interval start and end time are judged;Characteristic sequence applies disturbance, and consistency analysis is carried out, and event authenticity is judged;Synthesize subject sequence and consistency result, and output rotation monitoring information.The application fuses vision, control and seat and other multi-source features, constructs three-element relationship diagram and introduces topological reasoning and consistency determination, realizes the real identification and monitoring of double driver rotation process.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and vehicle-to-everything (V2X) safety monitoring technology, and in particular to a vehicle dual-driver rotation monitoring method and system based on multi-feature fusion. Background Technology

[0002] With increasingly stringent road transport safety regulations, dual-driver rotation systems are widely adopted in long-distance passenger transport, hazardous materials transport, and trunk logistics to reduce the risk of driver fatigue. Existing vehicle monitoring typically relies on in-vehicle cameras, dashcams, or driver monitoring systems to acquire cabin images, and combines these with a limited number of vehicle signals to identify driver on-duty status, fatigue, and distraction. Some solutions further utilize information such as seat occupancy and seatbelt status to assist in determining the status of the driver and co-driver, supporting rotation management and post-event traceability.

[0003] However, existing technologies mostly rely on single-modal or simple multimodal fusion to determine who the driver is or whether the seat has been changed, lacking accurate modeling of the complex relationships between the control subject, the seat subject, and behavioral changes. On the one hand, under conditions of occlusion, changes in lighting, or incomplete movements, vision-based face or posture recognition is prone to false positives and false negatives. On the other hand, relying solely on seat occupancy or seatbelt changes is insufficient to confirm whether the control subject has truly switched, and situations such as brief steering wheel flexion, assisted control, or no actual handover are easily misjudged as a complete handover, leading to unreliable handover conclusions.

[0004] Existing solutions generally lack effective differentiation of the authenticity of rotation events and the construction of verifiable evidence chains. They are difficult to accurately infer the actual transfer process of the driving entity in situations such as the same driver faking a seat change, two people interacting briefly, and the rotation process being complex and having an unstable time span. They are also difficult to output explanatory information that meets regulatory and accountability requirements.

[0005] Therefore, how to provide a vehicle dual-driver rotation monitoring method and system based on multi-feature fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a vehicle dual-driver rotation monitoring method and system based on multi-feature fusion. This invention constructs a correlation model between the driver, seat, and control behavior by fusing cockpit visual features, vehicle control features, and seat occupancy features. It then utilizes methods such as topology transfer reasoning, rotation intention potential energy analysis, and cross-modal consistency determination to achieve accurate monitoring and authenticity verification of the dual-driver rotation process. This invention effectively addresses the problems of inaccurate identification of the control subject and the susceptibility to misjudgment of the rotation process in complex cockpit environments. It can reliably identify genuine rotation behavior even under conditions of obstruction, short-term steering wheel intervention, and sham seat changes, generating interpretable rotation monitoring information. It possesses advantages such as high identification accuracy, strong anti-interference capability, and high regulatory reliability.

[0007] A vehicle dual-driver rotation monitoring method based on multi-feature fusion according to an embodiment of the present invention includes:

[0008] Multi-source data from the vehicle cabin is collected, and the multi-source data is preprocessed to obtain structured visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences.

[0009] The visual feature sequence is input into the improved RTMDet model to obtain the detection box information of the driver area, passenger area, hand area, steering wheel area, gear position area and seat area. Combined with the vehicle control feature sequence, feature data related to driving operation behavior is extracted.

[0010] Based on the detection box information and seat occupancy feature sequence, a ternary structure graph is constructed, edge connection relationships are established, and the topology transfer operator is used to perform topology transfer reasoning on the ternary structure graph at different times to obtain the control subject sequence, seat subject sequence and candidate rotation interval;

[0011] Based on candidate rotation intervals, combined with posture change characteristics, hand movement characteristics, seat occupancy change characteristics, and vehicle control characteristics, the potential energy of rotation intention is calculated to determine the rotation start and end time intervals.

[0012] Perturbations are applied to the visual feature sequence, seat occupancy feature sequence, and vehicle control feature sequence within the rotation start and end time interval to obtain the perturbed visual response sequence, seat response sequence, and vehicle control response sequence. Cross-modal consistency analysis is then performed to determine the authenticity of the rotation event.

[0013] Based on the sequence of the controlling entity and the sequence of the seating entity, the start and end time intervals of the rotation, and the authenticity determination results, a comprehensive judgment is made on the dual-driver rotation behavior to form dual-driver rotation monitoring information.

[0014] Optionally, the multi-source data includes vehicle cabin video data, vehicle control signal data, seat occupancy data, seat belt status data, and door status data.

[0015] Optionally, the preprocessing of multi-source data to obtain structured visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences includes:

[0016] The cockpit video data is subjected to time synchronization processing, image denoising processing, and inter-frame stabilization processing to form a visual feature sequence arranged in chronological order.

[0017] The vehicle control signal data is processed by time alignment, outlier removal, and feature extraction to form a vehicle control feature sequence arranged in chronological order.

[0018] The seat occupancy data is filtered, smoothed for instantaneous fluctuations, and extracted for occupancy status to form a seat occupancy feature sequence arranged in chronological order.

[0019] Optionally, the step of inputting the visual feature sequence into the improved RTMDet model to obtain detection box information for the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area includes:

[0020] Read the visual feature sequence, organize the visual feature sequence into time-by-time input, and at the same time read the vehicle control feature sequence that is time-aligned with the visual feature input unit at each time moment to form a one-to-one visual feature input and vehicle control feature input.

[0021] Visual features are input into an improved RTMDet model for detection. The improved RTMDet model includes a seat boundary guidance layer, an interaction contact relationship enhancement layer, and a mutual exclusion consistency constraint layer, which are connected in a cascaded manner.

[0022] The seat boundary guidance layer spatially constrains the detection features based on the prior knowledge of the fixed structure of the cockpit, divides the detection features into a feature subset of the main driver area and a feature subset of the co-driver area, and suppresses cross-regional candidate targets;

[0023] The interaction contact relationship enhancement layer encodes the spatial proximity and relative motion trends between the hand area and the steering wheel area and gear shift area, generating contact relationship features between the hand and the control components, and then injects these contact relationship features back into the detection features.

[0024] The mutual exclusion consistency constraint layer performs mutual exclusion correction on the detection confidence of the driver's area and the passenger's area at the same time, and outputs the detection result after mutual exclusion correction.

[0025] Based on the detection output of the improved RTMDet model, detection box information for the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area is obtained. The detection box information is then categorized and time-labeled to form a sequence of detection boxes arranged by time.

[0026] Based on the vehicle control feature sequence, feature data related to driving control behavior is extracted at each moment, and the feature data is time-labeled to form a control behavior feature sequence arranged by time.

[0027] The detection box sequence and the manipulation behavior feature sequence are aligned and fused according to time to generate structured input data.

[0028] Optionally, obtaining the control subject sequence, seat subject sequence, and candidate rotation interval includes:

[0029] Based on structured input data, a ternary structure graph is constructed at each time step, with driver nodes, seat nodes, and control behavior nodes as three types of nodes.

[0030] At each moment, the driver and seat occupancy relationship is established based on the driver detection frame, the passenger detection frame and the seat detection frame; the driver and control behavior are established based on the hand detection frame, the steering wheel detection frame and the gear detection frame; and the seat and control behavior are established based on the seat detection frame, the steering wheel detection frame and the gear detection frame, forming a ternary structure diagram sequence.

[0031] A topology transfer operator is introduced into the ternary structure graph sequence. The topology transfer operator includes local topology evolution units, physical constraint rule units, and global consistency correction units, wherein:

[0032] Local topological evolution units update the edge connections in the ternary structure graph between adjacent time steps;

[0033] The physical constraint rule unit restricts changes in the occupancy relationship edge and the control relationship edge to conform to the preset seat-changing rule and control subject switching rule;

[0034] The global consistency correction unit performs consistency checks and corrections on edge connection changes within a continuous time window.

[0035] The ternary structure graphs at different time points are inferred using the topology transfer operator to obtain the sequence of ternary structure graphs after topology transfer inference;

[0036] Based on the ternary structure graph sequence after topological transfer reasoning, at each time step, the identity of the controlling entity is determined according to the edge connection relationship between the driver and the control behavior, the identity of the seat entity is determined according to the edge connection relationship between the driver and the seat, and candidate rotation intervals are generated according to the time period when the controlling entity identity is stably switched from one driver to another.

[0037] Optionally, the calculation of the rotation intention potential energy and the determination of the rotation start and end time interval include:

[0038] Select candidate rotation intervals from the candidate rotation interval set, and sequentially read the posture features and hand action features in the visual feature sequence, the seat occupancy features in the seat occupancy feature sequence, and the vehicle control features in the vehicle control feature sequence on the time series corresponding to the candidate rotation intervals.

[0039] Based on posture features, the intensity value of posture change is calculated for each time point within the candidate rotation interval, which is used to characterize the degree of change of the driver's upper body posture relative to the reference posture at each time point.

[0040] Based on hand movement characteristics, seat occupancy characteristics, and vehicle control characteristics, the intensity values ​​of hand movement, seat occupancy change, and vehicle control change are calculated for each time point within the candidate rotation interval.

[0041] Within the candidate rotation interval, the intensity values ​​of posture change, hand movement, seat occupancy change, and vehicle control change at each moment are used as inputs. The posture potential energy components, hand potential energy components, seat potential energy components, and control potential energy components at each moment are obtained according to the preset weighting relationship. The potential energy components are then summed to form a rotation intention potential energy value sequence.

[0042] Time series analysis is performed on the potential energy value sequence of the rotation intention to determine the time period from the stable stage of potential energy value to a low value and then back to the stable stage. The start and end times of the time period are determined as the rotation start time and rotation end time, respectively, to obtain the rotation start and end time interval.

[0043] Optionally, the step of performing cross-modal consistency analysis to determine the authenticity of the rotation event includes:

[0044] Based on the determined start and end time intervals of the rotation, visual feature subsequences, seat occupancy feature subsequences, and vehicle control feature subsequences for the corresponding time periods are extracted from the visual feature sequence, seat occupancy feature sequence, and vehicle control feature sequence.

[0045] A visual perturbation input sequence is obtained by applying a perturbation of preset amplitude and direction to the visual feature subsequence, and a visual response sequence is generated based on the visual perturbation input sequence.

[0046] A preset perturbation is applied to the seat occupancy feature subsequence to obtain a seat perturbation input sequence and generate a seat response sequence; a preset perturbation is applied to the vehicle control feature subsequence to obtain a vehicle control perturbation input sequence and generate a vehicle control response sequence.

[0047] Within the rotation start and end time interval, cross-modal consistency analysis is performed on the visual response sequence, seat response sequence, and vehicle control response sequence. The direction, magnitude, and position of change of the three at each moment or within each time window are compared to obtain the corresponding cross-modal consistency analysis results.

[0048] The cross-modal consistency analysis results are compared with a preset consistency judgment threshold. When the cross-modal consistency meets the consistency judgment threshold, the rotation start and end time interval is marked as a real rotation event; otherwise, it is marked as a non-real rotation event, thus forming the authenticity judgment result of the rotation event.

[0049] Optionally, the step of comprehensively judging the dual-driver rotation behavior to form dual-driver rotation monitoring information includes:

[0050] Based on the sequence of the controlling entity and the sequence of the seat entity, the identity of the controlling entity, the identity of the seat entity, and the change status at each time point are extracted. The time period in which the identity remains consistent in consecutive time points is marked as the stable interval, and the time period in which the identity changes is marked as the switching interval.

[0051] The determined start and end time intervals of the rotation are matched with the switching intervals for time overlap. When the two partially or completely overlap on the time axis, the time interval is determined as a valid rotation interval, and the corresponding change direction of the control subject and the change direction of the seat subject are recorded.

[0052] The authenticity determination result of the rotation event is matched with the effective rotation interval. When the authenticity determination result is a real rotation event and an effective rotation interval exists, the interval is determined as the final dual-driver rotation event.

[0053] The start time, end time, change point of the control subject's identity, change path of the seat subject, and multimodal feature data within the corresponding time period are extracted from the final dual-driver rotation event. The information is then summarized to generate dual-driver rotation monitoring information.

[0054] The dual-driver rotation monitoring information is stored in the vehicle monitoring system, and the dual-driver rotation monitoring results, which include the actual rotation time interval, information on changes in the operating subject, information on changes in the seating subject, and related feature data, are output for real-time monitoring and recording of the dual-driver rotation status of the vehicle.

[0055] A vehicle dual-driver rotation monitoring system based on multi-feature fusion according to an embodiment of the present invention includes the following modules:

[0056] The data acquisition module is used to collect multi-source data from the vehicle cabin and preprocess it to generate visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences.

[0057] The detection module is used to input visual feature sequences into the improved RTMDet model and extract handling behavior features by combining them with vehicle control feature sequences.

[0058] The topology reasoning module is used to construct a ternary structure graph of driver, seat and control behavior and perform topology transfer reasoning to generate control subject sequence, seat subject sequence and candidate rotation interval;

[0059] The potential energy calculation module is used to calculate the potential energy of the rotation intention based on the candidate rotation interval and determine the start and end time interval of the rotation.

[0060] The consistency analysis module is used to apply perturbations to the multimodal features within the rotation start and end time interval and perform cross-modal consistency analysis to determine the authenticity judgment result of the rotation event.

[0061] The rotation determination module is used to generate dual-driver rotation monitoring information based on the control subject sequence, seat subject sequence, rotation start and end time interval, and authenticity determination results.

[0062] The beneficial effects of this invention are:

[0063] This invention unifies and structures the preprocessing of multi-source data, including cockpit video, vehicle control signals, and seat occupancy, and inputs the visual feature sequences into an improved RTMDet model to obtain bounding box information for key areas such as the driver's seat, passenger seat, hands, steering wheel, gear shift, and seat. This information is then combined with vehicle control feature sequences to extract control-related features, thereby improving the perceptual completeness and temporal alignment consistency of key cockpit entities and key motion elements from the source. Compared to solutions relying solely on single visual recognition or single seat state judgment, this invention maintains stable element acquisition capabilities even in complex environments such as occlusion, lighting changes, and incomplete motion, providing a reliable input foundation for subsequent rotational inference.

[0064] This invention further constructs a ternary structure graph of driver, seat, and control behavior based on detection box information and seat occupancy feature sequences. It then utilizes a topology transfer operator to perform continuous transfer reasoning on the ternary relationship over time, forming a control subject sequence, a seat subject sequence, and candidate rotation intervals. This elevates rotation recognition from single-frame judgment to a holistic inference of the relationship's evolution over time. This mechanism can uniformly model changes in the control subject and seat subject, effectively reducing instantaneous misjudgments caused by short-term interactions and assisted control between the driver and passenger, making the recognition of the driver's transfer process more consistent with real driving behavior patterns and physical constraints.

[0065] Based on candidate rotation intervals, this invention determines the start and end time intervals of rotation through rotation intention potential energy calculation. It then applies perturbations to visual, seat occupancy, and vehicle control characteristics within this interval to generate a multimodal response sequence. Based on cross-modal consistency analysis, it forms authenticity determination information, thereby distinguishing between genuine and spurious rotations. After comprehensive judgment, it generates dual-driver rotation monitoring information. This invention not only provides rotation time intervals and control entity switching information but also suppresses spurious rotation identification in scenarios such as short-term steering wheel intervention and feigned seat switching. This improves the verifiability and regulatory reliability of rotation determination, meeting the safety management and post-event traceability needs of dual-driver operation scenarios. Attached Figure Description

[0066] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0067] Figure 1 This is a flowchart of a vehicle dual-driver rotation monitoring method based on multi-feature fusion proposed in this invention;

[0068] Figure 2 This is a schematic diagram of the structure of a vehicle dual-driver rotation monitoring system based on multi-feature fusion proposed in this invention. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0070] refer to Figure 1 A vehicle dual-driver rotation monitoring method based on multi-feature fusion includes:

[0071] Multi-source data from the vehicle cabin is collected, and the multi-source data is preprocessed to obtain structured visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences.

[0072] The visual feature sequence is input into the improved RTMDet model to obtain the detection box information of the driver area, passenger area, hand area, steering wheel area, gear position area and seat area. Combined with the vehicle control feature sequence, feature data related to driving operation behavior is extracted.

[0073] Based on the detection box information and seat occupancy feature sequence, a ternary structure graph is constructed, edge connection relationships are established, and the topology transfer operator is used to perform topology transfer reasoning on the ternary structure graph at different times to obtain the control subject sequence, seat subject sequence and candidate rotation interval;

[0074] Based on candidate rotation intervals, combined with posture change characteristics, hand movement characteristics, seat occupancy change characteristics, and vehicle control characteristics, the potential energy of rotation intention is calculated to determine the rotation start and end time intervals.

[0075] Perturbations are applied to the visual feature sequence, seat occupancy feature sequence, and vehicle control feature sequence within the rotation start and end time interval to obtain the perturbed visual response sequence, seat response sequence, and vehicle control response sequence. Cross-modal consistency analysis is then performed to determine the authenticity of the rotation event.

[0076] Based on the sequence of the controlling entity and the sequence of the seating entity, the start and end time intervals of the rotation, and the authenticity determination results, a comprehensive judgment is made on the dual-driver rotation behavior to form dual-driver rotation monitoring information.

[0077] In this embodiment, the multi-source data includes vehicle cabin video data, vehicle control signal data, seat occupancy data, seat belt status data, and door status data.

[0078] In this embodiment, the preprocessing of multi-source data to obtain structured visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences includes:

[0079] The cockpit video data is subjected to time synchronization processing, image denoising processing, and inter-frame stabilization processing to form a visual feature sequence arranged in chronological order.

[0080] The vehicle control signal data is processed by time alignment, outlier removal, and feature extraction to form a vehicle control feature sequence arranged in chronological order.

[0081] The seat occupancy data is filtered, smoothed for instantaneous fluctuations, and extracted for occupancy status to form a seat occupancy feature sequence arranged in chronological order.

[0082] In this embodiment, the step of inputting the visual feature sequence into the improved RTMDet model to obtain the detection box information of the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area includes:

[0083] Read the visual feature sequence, organize the visual feature sequence into time-by-time input, and at the same time read the vehicle control feature sequence that is time-aligned with the visual feature input unit at each time moment to form a one-to-one visual feature input and vehicle control feature input.

[0084] Visual features are input into an improved RTMDet model for detection. The improved RTMDet model includes a seat boundary guidance layer, an interaction contact relationship enhancement layer, and a mutual exclusion consistency constraint layer, which are connected in a cascaded manner.

[0085] The seat boundary guidance layer spatially constrains the detection features based on the prior knowledge of the fixed structure of the cockpit, divides the detection features into a feature subset of the main driver area and a feature subset of the co-driver area, and suppresses cross-regional candidate targets;

[0086] The interaction contact relationship enhancement layer encodes the spatial proximity and relative motion trends between the hand area and the steering wheel area and gear shift area, generating contact relationship features between the hand and the control components, and then injects these contact relationship features back into the detection features.

[0087] The mutual exclusion consistency constraint layer performs mutual exclusion correction on the detection confidence of the driver's area and the passenger's area at the same time, and outputs the detection result after mutual exclusion correction.

[0088] Based on the detection output of the improved RTMDet model, detection box information for the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area is obtained. The detection box information is then categorized and time-labeled to form a sequence of detection boxes arranged by time.

[0089] Based on the vehicle control feature sequence, feature data related to driving control behavior is extracted at each moment, and the feature data is time-labeled to form a control behavior feature sequence arranged by time.

[0090] The detection box sequence and the manipulation behavior feature sequence are aligned and fused according to time to generate structured input data.

[0091] In this embodiment, obtaining the control subject sequence, the seat subject sequence, and the candidate rotation interval includes:

[0092] Based on structured input data, a ternary structure graph is constructed at each time step, with driver nodes, seat nodes, and control behavior nodes as three types of nodes.

[0093] At each moment, the driver and seat occupancy relationship is established based on the driver detection frame, the passenger detection frame and the seat detection frame; the driver and control behavior are established based on the hand detection frame, the steering wheel detection frame and the gear detection frame; and the seat and control behavior are established based on the seat detection frame, the steering wheel detection frame and the gear detection frame, forming a ternary structure diagram sequence.

[0094] A topology transfer operator is introduced into the ternary structure graph sequence. The topology transfer operator includes local topology evolution units, physical constraint rule units, and global consistency correction units, wherein:

[0095] Local topological evolution units update the edge connections in the ternary structure graph between adjacent time steps;

[0096] The physical constraint rule unit restricts changes in occupancy and control relationships to conform to preset seat-changing and control subject switching rules, wherein:

[0097] The preset seat-changing rules are as follows: within any consecutive time window, the occupancy status of the driver's seat and the passenger seat should transition once from driver's seat occupied and passenger seat occupied to driver's seat vacant and passenger seat occupied, or from driver's seat occupied and passenger seat occupied to driver's seat occupied and passenger seat vacant, and remain stable after the transition is completed; occupancy relationship edges are only allowed to migrate in the manner of gradually weakening and dissolving the connection relationship between the driver node and the driver's seat node, while gradually strengthening and establishing the connection relationship between the driver node and the passenger seat node, or in the opposite direction, and the same driver node is only allowed to maintain an occupancy relationship edge with only one seat node at any given time;

[0098] The switching rules for control entities are as follows: Within any consecutive time window, the control relationship edges corresponding to steering wheel control, pedal control, and gear shift control should undergo a continuous migration from the original control entity driver node to the target control entity driver node. This migration process is characterized by a continuous weakening of the association strength between the original control entity and the control behavior node, and a continuous strengthening of the association strength between the target control entity and the control behavior node, which remains stable after the migration is completed. Multiple back-and-forth switching of control relationship edges is not allowed within the same time window, and two driver nodes are not allowed to simultaneously form a high-intensity control relationship with the same control behavior node.

[0099] The global consistency correction unit performs consistency checks and corrections on edge connection changes within a continuous time window.

[0100] The ternary structure graphs at different time points are inferred using the topology transfer operator to obtain the sequence of ternary structure graphs after topology transfer inference;

[0101] Based on the ternary structure graph sequence after topological transfer reasoning, at each time step, the identity of the controlling entity is determined according to the edge connection relationship between the driver and the control behavior, the identity of the seat entity is determined according to the edge connection relationship between the driver and the seat, and candidate rotation intervals are generated according to the time period when the controlling entity identity is stably switched from one driver to another.

[0102] In this embodiment, calculating the potential energy of the rotation intention and determining the rotation start and end time interval includes:

[0103] Select candidate rotation intervals from the candidate rotation interval set, and sequentially read the posture features and hand action features in the visual feature sequence, the seat occupancy features in the seat occupancy feature sequence, and the vehicle control features in the vehicle control feature sequence on the time series corresponding to the candidate rotation intervals.

[0104] Based on attitude features, attitude change intensity values ​​are calculated for each time step within the candidate rotation interval. These values ​​characterize the degree of change in the driver's upper body attitude relative to the reference attitude at each time step. Specifically, the calculation of attitude change intensity values ​​for each time step within the candidate rotation interval is as follows:

[0105] The upper body posture parameters are determined based on the posture characteristics at each time point, including head position parameters, shoulder position parameters and torso orientation parameters, and the upper body posture parameters at the start time of the candidate rotation interval or the preset stable driving time are selected as reference posture parameters.

[0106] The upper body posture parameters at each moment are compared with the reference posture parameters to obtain the head displacement, shoulder displacement and torso orientation change, and the difference is normalized.

[0107] The normalized head displacement, shoulder displacement, and torso orientation change are fused according to preset weights to output the posture change intensity value at the corresponding moment.

[0108] Based on hand movement characteristics, seat occupancy characteristics, and vehicle control characteristics, the intensity values ​​of hand movement, seat occupancy change, and vehicle control change are calculated at each time point within the candidate rotation interval. Specifically, the calculation of these values ​​at each time point within the candidate rotation interval is as follows:

[0109] Based on the hand movement characteristics at each moment, the position parameters and motion state parameters of the hand relative to the steering wheel and gear position are determined. The hand position and motion state at the beginning of the candidate rotation interval or the preset stable driving moment are used as reference states. The hand position change and motion amplitude at each moment are calculated differentially, and the differential results are normalized to obtain the hand movement intensity value at the corresponding moment.

[0110] Based on the seat occupancy characteristics at each time point, the occupancy status parameters and pressure distribution parameters of the driver's seat and the passenger seat are extracted. Using the seat occupancy status at a reference time as a benchmark, the changes in seat occupancy status and pressure distribution at each time point are calculated, and the changes are normalized to obtain the seat occupancy change intensity value at the corresponding time point.

[0111] Based on the vehicle control characteristics at each time point, the parameters of steering wheel angle change, throttle opening change, and brake opening change are extracted. Using the vehicle control state at the reference time as a benchmark, the change amplitude of the control parameters at each time point is calculated, and the change amplitude is normalized and fused to obtain the vehicle control change intensity value at the corresponding time point.

[0112] Within the candidate rotation interval, the intensity values ​​of posture change, hand movement, seat occupancy change, and vehicle control change at each moment are used as inputs. The posture potential energy component, hand potential energy component, seat potential energy component, and control potential energy component at each moment are obtained according to a preset weighting relationship. These potential energy components are then summed to form a sequence of rotation intention potential energy values. The preset weighting relationship is as follows:

[0113] The posture potential energy component corresponding to the posture change intensity value is assigned a first weight. The first weight is used to reflect the dominant role of the driver's upper body posture change in the rotation process. Its weight is higher than that of a single local action but lower than that of the seat occupancy change.

[0114] The hand potential energy component corresponding to the hand movement intensity value is assigned a second weight. The second weight is used to reflect the influence of changes in hand contact with control components such as the steering wheel and gear shift on the shifting intention. The weight is lower than the weight of the seat potential energy component and is on the same order of magnitude as the weight of the posture potential energy component.

[0115] The seat potential energy component corresponding to the seat occupancy change intensity value is assigned a third weight, which is used to reflect the decisive influence of the driver's actual body position shift on the rotation intention. The weight is the highest weight in the preset weighting relationship. The control potential energy component corresponding to the vehicle control change intensity value is assigned a fourth weight, which is used to reflect the auxiliary constraint effect of the change in driving behavior on the rotation intention. The weight is less than the first weight, the second weight, and the third weight.

[0116] Time series analysis is performed on the potential energy value sequence of the rotation intention to determine the time period from the stable stage of potential energy value to a low value and then back to the stable stage. The start and end times of the time period are determined as the rotation start time and rotation end time, respectively, to obtain the rotation start and end time interval.

[0117] In this embodiment, the step of performing cross-modal consistency analysis to determine the authenticity of the rotation event includes:

[0118] Based on the determined start and end time intervals of the rotation, visual feature subsequences, seat occupancy feature subsequences, and vehicle control feature subsequences for the corresponding time periods are extracted from the visual feature sequence, seat occupancy feature sequence, and vehicle control feature sequence.

[0119] A visual perturbation input sequence is obtained by applying a perturbation of preset amplitude and direction to the visual feature subsequence, and a visual response sequence is generated based on the visual perturbation input sequence.

[0120] A preset perturbation is applied to the seat occupancy feature subsequence to obtain a seat perturbation input sequence and generate a seat response sequence; a preset perturbation is applied to the vehicle control feature subsequence to obtain a vehicle control perturbation input sequence and generate a vehicle control response sequence.

[0121] Within the rotation start and end time interval, cross-modal consistency analysis is performed on the visual response sequence, seat response sequence, and vehicle control response sequence. The direction, magnitude, and position of change of the three at each moment or within each time window are compared to obtain the corresponding cross-modal consistency analysis results.

[0122] The cross-modal consistency analysis results are compared with a preset consistency judgment threshold. When the cross-modal consistency meets the consistency judgment threshold, the rotation start and end time interval is marked as a real rotation event; otherwise, it is marked as a non-real rotation event, thus forming the authenticity judgment result of the rotation event.

[0123] In this embodiment, the step of comprehensively judging the dual-driver rotation behavior to form dual-driver rotation monitoring information includes:

[0124] Based on the sequence of the controlling entity and the sequence of the seat entity, the identity of the controlling entity, the identity of the seat entity, and the change status at each time point are extracted. The time period in which the identity remains consistent in consecutive time points is marked as the stable interval, and the time period in which the identity changes is marked as the switching interval.

[0125] The determined start and end time intervals of the rotation are matched with the switching intervals for time overlap. When the two partially or completely overlap on the time axis, the time interval is determined as a valid rotation interval, and the corresponding change direction of the control subject and the change direction of the seat subject are recorded.

[0126] The authenticity determination result of the rotation event is matched with the effective rotation interval. When the authenticity determination result is a real rotation event and an effective rotation interval exists, the interval is determined as the final dual-driver rotation event.

[0127] The start time, end time, change point of the control subject's identity, change path of the seat subject, and multimodal feature data within the corresponding time period are extracted from the final dual-driver rotation event. The information is then summarized to generate dual-driver rotation monitoring information.

[0128] The dual-driver rotation monitoring information is stored in the vehicle monitoring system, and the dual-driver rotation monitoring results, which include the actual rotation time interval, information on changes in the operating subject, information on changes in the seating subject, and related feature data, are output for real-time monitoring and recording of the dual-driver rotation status of the vehicle.

[0129] refer to Figure 2 A vehicle dual-driver rotation monitoring system based on multi-feature fusion includes the following modules:

[0130] The data acquisition module is used to collect multi-source data from the vehicle cabin and preprocess it to generate visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences.

[0131] The detection module is used to input visual feature sequences into the improved RTMDet model and extract handling behavior features by combining them with vehicle control feature sequences.

[0132] The topology reasoning module is used to construct a ternary structure graph of driver, seat and control behavior and perform topology transfer reasoning to generate control subject sequence, seat subject sequence and candidate rotation interval;

[0133] The potential energy calculation module is used to calculate the potential energy of the rotation intention based on the candidate rotation interval and determine the start and end time interval of the rotation.

[0134] The consistency analysis module is used to apply perturbations to the multimodal features within the rotation start and end time interval and perform cross-modal consistency analysis to determine the authenticity judgment result of the rotation event.

[0135] The rotation determination module is used to generate dual-driver rotation monitoring information based on the control subject sequence, seat subject sequence, rotation start and end time interval, and authenticity determination results.

[0136] Example 1:

[0137] To verify the feasibility of this invention in practice, it was applied to heavy-duty freight vehicles of a logistics company. This company mainly undertakes cross-regional transportation of daily necessities, with one-way transport distances ranging from 380 to 650 kilometers and a single trip taking approximately 6 to 9 hours. The vehicles are equipped with two drivers according to management requirements, and the two drivers rotate during service areas or at low speeds. In actual operation, situations such as drivers providing brief assistance with the steering wheel, leaning forward while conversing, low light at night, and partial obstruction make accurate monitoring of the driver rotation difficult.

[0138] In this scenario, each vehicle is equipped with a cockpit-facing camera, a vehicle control signal acquisition device, and driver and passenger seat occupancy sensors. All devices are connected to the vehicle dual-driver rotation monitoring system based on multi-feature fusion described in this invention. During vehicle operation, the system continuously collects cockpit video data, steering wheel angle, throttle and brake opening, gear status, and seat occupancy status. It performs time synchronization and preprocessing on the multi-source data to generate structured visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences.

[0139] The system then inputs the visual feature sequence into the improved RTMDet model to detect the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area, and extracts features related to driving behavior by combining the vehicle control feature sequence. In actual operation, even at night or when the driver's body is partially obscured, the system can still stably output detection box information, providing continuous input for subsequent inference.

[0140] The system constructs a ternary structure graph of driver, seat, and control behavior based on detection box information and seat occupancy feature sequences. It then uses a topology transfer operator to perform topology transfer inference on the ternary structure graph at different times, generating control subject sequences, seat subject sequences, and candidate switching intervals. For cases involving only short-term steering wheel manipulation or assisted observation behavior, topology transfer inference does not generate stable control subject and seat subject transitions, thus avoiding the erroneous generation of switching intervals.

[0141] When the system detects a continuous migration trend between the controlling subject and the seated subject over time, it generates candidate rotation intervals. Combining posture change features, hand gesture features, seat occupancy change features, and vehicle control features, it calculates the potential energy of the rotation intention to determine the start and end time intervals of the rotation. This step effectively suppresses instantaneous fluctuations caused by short-term behavior, making the rotation time positioning more stable.

[0142] Subsequently, the system applies perturbations to the visual feature sequences, seat occupancy feature sequences, and vehicle control feature sequences within the rotation start and end time interval, generating perturbed multimodal response sequences and performing cross-modal consistency analysis. When changes in visual posture, seat occupancy, and control behavior are consistent over time, the system determines it as a genuine rotation event; when only a single modality changes, it is determined as a non-genuine rotation.

[0143] Ultimately, the system comprehensively assesses dual-driver rotation behavior based on the sequence of controlling entities, the sequence of seating entities, the start and end time intervals of the rotation, and the authenticity determination results. This results in dual-driver rotation monitoring information, which is then uploaded to the company's vehicle management platform for daily safety management and post-event verification.

[0144] Table 1. Comparison of Monitoring Effects of Dual-Driver Rotation

[0145] Rotation event recognition count 214 226 219 212 True rotation recognition accuracy 100% 90.7% 93.9% 96.3% False rotation misjudgment rate 0% 8.4% 5.1% 2.8% Short-term support error rate 0% 6.9% 4.3% 1.9% Average error in rotation start and end times (seconds) 0 5.8 4.2 3.1 Nighttime scene stability determination rate 100% 88.5% 91.6% 95.4%

[0146] Table 1 shows that, in terms of the number of rotation events identified and the accuracy of actual rotation recognition, both the vision-only solution and the vision-plus-seat solution identified more rotation events than the manual verification results. This indicates that in actual operation, there are instances where short-term handrail manipulation or assisted operation behaviors are misjudged as rotation events, with the vision-only solution being more significantly affected. The method of this invention identifies the number of rotation events closest to the manual verification results, and its actual rotation recognition accuracy reaches 96.3%, significantly better than the comparative solutions, demonstrating better stability and consistency in the overall determination of rotation events.

[0147] Regarding the false rotation misjudgment rate and the short-term steering wheel misjudgment rate, only the visual solution had a relatively high misjudgment level. Although the visual + seat solution showed some improvement, it still struggled to completely distinguish between genuine rotation and short-term interaction behaviors. The method of this invention, through multi-feature fusion and topology transfer reasoning, reduced the false rotation misjudgment rate and the short-term steering wheel misjudgment rate to 2.8% and 1.9%, respectively, effectively suppressing the interference of non-genuine rotation behaviors on the monitoring results and demonstrating a stronger ability to distinguish complex driving behaviors.

[0148] Regarding the positioning of rotation start and end times and stability in nighttime scenarios, the method of this invention controls the average error of rotation start and end times to around 3 seconds, significantly less than the comparative scheme, and maintains a stability judgment rate of over 95% even under low-light conditions at night. In complex environments, this invention can not only accurately identify whether a rotation has occurred, but also precisely characterize the time interval of the rotation, meeting the accuracy and reliability requirements of actual operational supervision.

[0149] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A vehicle dual-driver rotation monitoring method based on multi-feature fusion, characterized in that, include: Multi-source data from the vehicle cabin is collected, and the multi-source data is preprocessed to obtain structured visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences. The visual feature sequence is input into the improved RTMDet model to obtain bounding box information for the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area. Combined with the vehicle control feature sequence, feature data related to driving behavior is extracted, including: The improved RTMDet model includes a seat boundary guidance layer, an interaction contact relationship enhancement layer, and a mutual exclusion consistency constraint layer, which are connected in a cascaded manner. The seat boundary guidance layer spatially constrains the detection features based on the prior knowledge of the fixed structure of the cockpit, divides the detection features into a feature subset of the main driver area and a feature subset of the co-driver area, and suppresses cross-regional candidate targets; The interaction contact relationship enhancement layer encodes the spatial proximity and relative motion trends between the hand area and the steering wheel area and gear shift area, generating contact relationship features between the hand and the control components, and then injects these contact relationship features back into the detection features. The mutual exclusion consistency constraint layer performs mutual exclusion correction on the detection confidence of the driver's area and the passenger's area at the same time, and outputs the detection result after mutual exclusion correction. Based on the detection box information and seat occupancy feature sequence, a ternary structure graph is constructed, edge connection relationships are established, and the topology transfer operator is used to perform topology transfer reasoning on the ternary structure graph at different times to obtain the control subject sequence, seat subject sequence and candidate rotation interval, where the driver node, seat node and control behavior node are the three types of nodes in the ternary structure graph. Based on candidate rotation intervals, combined with posture change characteristics, hand movement characteristics, seat occupancy change characteristics, and vehicle control characteristics, the potential energy of rotation intention is calculated to determine the rotation start and end time intervals. Perturbations are applied to the visual feature sequence, seat occupancy feature sequence, and vehicle control feature sequence within the rotation start and end time interval to obtain the perturbed visual response sequence, seat response sequence, and vehicle control response sequence. Cross-modal consistency analysis is then performed to determine the authenticity of the rotation event. Based on the sequence of the controlling entity and the sequence of the seating entity, the start and end time intervals of the rotation, and the authenticity determination results, a comprehensive judgment is made on the dual-driver rotation behavior to form dual-driver rotation monitoring information.

2. The vehicle dual-driver rotation monitoring method based on multi-feature fusion according to claim 1, characterized in that, The multi-source data includes vehicle cabin video data, vehicle control signal data, seat occupancy data, seat belt status data, and door status data.

3. The vehicle dual-driver rotation monitoring method based on multi-feature fusion according to claim 2, characterized in that, The preprocessing of multi-source data yields structured visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences, including: The cockpit video data is subjected to time synchronization processing, image denoising processing, and inter-frame stabilization processing to form a visual feature sequence arranged in chronological order. The vehicle control signal data is processed by time alignment, outlier removal, and feature extraction to form a vehicle control feature sequence arranged in chronological order. The seat occupancy data is filtered, smoothed for instantaneous fluctuations, and extracted for occupancy status to form a seat occupancy feature sequence arranged in chronological order.

4. The vehicle dual-driver rotation monitoring method based on multi-feature fusion according to claim 1, characterized in that, The step of inputting visual feature sequences into the improved RTMDet model to obtain detection box information for the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area includes: Read the visual feature sequence, organize the visual feature sequence into time-by-time input, and at the same time read the vehicle control feature sequence that is time-aligned with the visual feature input unit at each time moment to form a one-to-one visual feature input and vehicle control feature input. Visual features are input into the improved RTMDet model for detection: Based on the detection output of the improved RTMDet model, detection box information for the driver's area, passenger's area, hand area, steering wheel area, gear shift area, and seat area is obtained. The detection box information is then categorized and time-labeled to form a sequence of detection boxes arranged by time. Based on the vehicle control feature sequence, feature data related to driving control behavior is extracted at each moment, and the feature data is time-labeled to form a control behavior feature sequence arranged by time. The detection box sequence and the manipulation behavior feature sequence are aligned and fused according to time to generate structured input data.

5. The vehicle dual-driver rotation monitoring method based on multi-feature fusion according to claim 1, characterized in that, The process of obtaining the control subject sequence, seat subject sequence, and candidate rotation interval includes: Based on structured input data, a ternary structure graph is constructed at each time step; At each moment, the driver and seat occupancy relationship is established based on the driver detection frame, the passenger detection frame and the seat detection frame; the driver and control behavior are established based on the hand detection frame, the steering wheel detection frame and the gear detection frame; and the seat and control behavior are established based on the seat detection frame, the steering wheel detection frame and the gear detection frame, forming a ternary structure diagram sequence. A topology transfer operator is introduced into the ternary structure graph sequence. The topology transfer operator includes local topology evolution units, physical constraint rule units, and global consistency correction units, wherein: Local topological evolution units update the edge connections in the ternary structure graph between adjacent time steps; The physical constraint rule unit restricts changes in the occupancy relationship edge and the control relationship edge to conform to the preset seat-changing rule and control subject switching rule; The global consistency correction unit performs consistency checks and corrections on edge connection changes within a continuous time window. The ternary structure graphs at different time points are inferred using the topology transfer operator to obtain the sequence of ternary structure graphs after topology transfer inference; Based on the ternary structure graph sequence after topological transfer reasoning, at each time step, the identity of the controlling entity is determined according to the edge connection relationship between the driver and the control behavior, the identity of the seat entity is determined according to the edge connection relationship between the driver and the seat, and candidate rotation intervals are generated according to the time period when the controlling entity identity is stably switched from one driver to another.

6. The vehicle dual-driver rotation monitoring method based on multi-feature fusion according to claim 1, characterized in that, The calculation of the rotation intention potential energy and the determination of the rotation start and end time intervals include: Select candidate rotation intervals from the candidate rotation interval set, and sequentially read the posture features and hand action features in the visual feature sequence, the seat occupancy features in the seat occupancy feature sequence, and the vehicle control features in the vehicle control feature sequence on the time series corresponding to the candidate rotation intervals. Based on posture features, the intensity value of posture change is calculated for each time point within the candidate rotation interval, which is used to characterize the degree of change of the driver's upper body posture relative to the reference posture at each time point. Based on hand movement characteristics, seat occupancy characteristics, and vehicle control characteristics, the intensity values ​​of hand movement, seat occupancy change, and vehicle control change are calculated for each time point within the candidate rotation interval. Within the candidate rotation interval, the intensity values ​​of posture change, hand movement, seat occupancy change, and vehicle control change at each moment are used as inputs. The posture potential energy components, hand potential energy components, seat potential energy components, and control potential energy components at each moment are obtained according to the preset weighting relationship. The potential energy components are then summed to form a rotation intention potential energy value sequence. Time series analysis is performed on the potential energy value sequence of the rotation intention to determine the time period from the stable stage of potential energy value to a low value and then back to the stable stage. The start and end times of the time period are determined as the rotation start time and rotation end time, respectively, to obtain the rotation start and end time interval.

7. The vehicle dual-driver rotation monitoring method based on multi-feature fusion according to claim 1, characterized in that, The process of performing cross-modal consistency analysis to determine the authenticity of rotation events includes: Based on the determined start and end time intervals of the rotation, visual feature subsequences, seat occupancy feature subsequences, and vehicle control feature subsequences for the corresponding time periods are extracted from the visual feature sequence, seat occupancy feature sequence, and vehicle control feature sequence. A visual perturbation input sequence is obtained by applying a perturbation of preset amplitude and direction to the visual feature subsequence, and a visual response sequence is generated based on the visual perturbation input sequence. A preset perturbation is applied to the seat occupancy feature subsequence to obtain a seat perturbation input sequence and generate a seat response sequence; a preset perturbation is applied to the vehicle control feature subsequence to obtain a vehicle control perturbation input sequence and generate a vehicle control response sequence. Within the rotation start and end time interval, cross-modal consistency analysis is performed on the visual response sequence, seat response sequence, and vehicle control response sequence. The direction, magnitude, and position of change of the three at each moment or within each time window are compared to obtain the corresponding cross-modal consistency analysis results. The cross-modal consistency analysis results are compared with a preset consistency judgment threshold. When the cross-modal consistency meets the consistency judgment threshold, the rotation start and end time interval is marked as a real rotation event; otherwise, it is marked as a non-real rotation event, thus forming the authenticity judgment result of the rotation event.

8. The vehicle dual-driver rotation monitoring method based on multi-feature fusion according to claim 1, characterized in that, The comprehensive judgment of dual-driver rotation behavior to form dual-driver rotation monitoring information includes: Based on the sequence of the controlling entity and the sequence of the seat entity, the identity of the controlling entity, the identity of the seat entity, and the change status at each time point are extracted. The time period in which the identity remains consistent in consecutive time points is marked as the stable interval, and the time period in which the identity changes is marked as the switching interval. The determined start and end time intervals of the rotation are matched with the switching intervals for time overlap. When the two partially or completely overlap on the time axis, the time interval is determined as a valid rotation interval, and the corresponding change direction of the control subject and the change direction of the seat subject are recorded. The authenticity determination result of the rotation event is matched with the effective rotation interval. When the authenticity determination result is a real rotation event and an effective rotation interval exists, the interval is determined as the final dual-driver rotation event. The start time, end time, change point of the control subject's identity, change path of the seat subject, and multimodal feature data within the corresponding time period are extracted from the final dual-driver rotation event. The information is then summarized to generate dual-driver rotation monitoring information. The dual-driver rotation monitoring information is stored in the vehicle monitoring system, and the dual-driver rotation monitoring results, which include the actual rotation time interval, information on changes in the operating subject, information on changes in the seating subject, and related feature data, are output for real-time monitoring and recording of the dual-driver rotation status of the vehicle.

9. A vehicle dual-driver rotation monitoring system based on multi-feature fusion, wherein the vehicle dual-driver rotation monitoring method based on multi-feature fusion as described in any one of claims 1 to 8 is characterized in that, Includes the following modules: The data acquisition module is used to collect multi-source data from the vehicle cabin and preprocess it to generate visual feature sequences, vehicle control feature sequences, and seat occupancy feature sequences. The detection module is used to input visual feature sequences into the improved RTMDet model and extract handling behavior features by combining them with vehicle control feature sequences. The topology reasoning module is used to construct a ternary structure graph of driver, seat and control behavior and perform topology transfer reasoning to generate control subject sequence, seat subject sequence and candidate rotation interval; The potential energy calculation module is used to calculate the potential energy of the rotation intention based on the candidate rotation interval and determine the start and end time interval of the rotation. The consistency analysis module is used to apply perturbations to the multimodal features within the rotation start and end time interval and perform cross-modal consistency analysis to determine the authenticity judgment result of the rotation event. The rotation determination module is used to generate dual-driver rotation monitoring information based on the control subject sequence, seat subject sequence, rotation start and end time interval, and authenticity determination results.

Images