A human-machine co-driving method and device for drivers with upper limb disabilities

Abstract

The present invention belongs to the field of autonomous driving testing technology, specifically a human-machine co-driving method and device for drivers with upper limb disabilities. The device includes drive and brake pedals, a vehicle control unit (VCU), a steering mechanism, a steering control pedal, a bracket, a rotating column, screws, first and second pedals, a spring base, first and second return springs, first and second bases, first and second Hall effect sensors, a protective shell, first and second magnets, a housing, and a thrust washer. The co-driving method of the present invention uses the right foot to control the accelerator and brake pedals, and the left foot to control the steering pedal, converting pedal position and speed change signals into digital signals to obtain driver control commands. Simultaneously, based on the surrounding driving environment information obtained by on-board sensors, a vehicle lateral control signal is calculated and output. Finally, through the allocation of human-machine driving rights, a final control command is output to control the steering execution unit, drive execution unit, and brake execution unit to achieve vehicle control.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving testing, and specifically provides a human-machine co-driving method and device for drivers with upper limb disabilities. Background Art

[0002] Upper limb disability can hinder drivers' ability to control the steering wheel, posing a significant challenge to their daily commuting. With advances in autonomous driving technology, the ability for drivers with upper limb disabilities to travel by car will become a reality. However, drivers with upper limb disabilities still lack direct control of the steering wheel, requiring the autonomous driving system to provide lateral control of the vehicle. Therefore, it is crucial to effectively integrate drivers with autonomous driving systems to control the vehicle's lateral and longitudinal movements. Furthermore, to ensure a superior driving experience for drivers with upper limb disabilities, the development of assisted driving devices for steering wheel control is crucial. With advances in drive-by-wire technology, electronic signals replace traditional mechanical connections to achieve precise vehicle control, which will play a greater role in future autonomous driving.

[0003] Patents for human-machine co-driving systems and devices for upper limb disabled drivers in the mobility sector are relatively rare. While these systems can help disabled drivers accomplish their driving tasks, they still place a heavy burden on drivers. For example, currently, there are no human-machine co-driving systems specifically designed for upper limb disabled drivers; only assisted driving devices exist. This prevents the full utilization of the convenience offered by autonomous driving systems. Therefore, developing human-machine co-driving systems for upper limb disabled drivers is of great significance. Summary of the Invention

[0004] The present invention provides a human-machine co-driving method and device for drivers with upper limb disabilities. The right foot controls the accelerator and brake pedals, while the left foot controls the steering pedal. Pedal position and speed change signals are converted into digital signals to obtain driver control commands. Simultaneously, based on ambient driving environment information obtained by onboard sensors, a lateral vehicle control signal is calculated and output. Finally, through human-machine driving rights allocation, the final control command is output to control the steering, drive, and brake actuators to achieve vehicle control. This solves the problem of upper limb disabled drivers being unable to steer the vehicle using two-handed steering.

[0005] The technical solution of the present invention is described as follows in conjunction with the accompanying drawings:

[0006] In a first aspect, the present invention provides a human-machine co-driving method for a driver with upper limb disabilities, comprising the following steps:

[0007] Drivers with upper limb disabilities use their right foot to control the longitudinal control device. The electronic sensor senses the depth of the pedal depression and converts the pedal depth into an electrical signal to generate the driver's acceleration and braking control signals. The left foot controls the lateral control device to generate the vehicle steering control signal.

[0008] The vehicle's onboard sensors, including cameras, millimeter-wave radars, lidars, ultrasonic radars, and inertial navigation systems, perceive the surrounding driving environment. The autonomous driving decision-making system, based on a multimodal large model, outputs lateral and longitudinal control signals a2.

[0009] Human-machine co-driving realizes three driving modes: driver, automatic driving and human-machine co-driving; when the driver wants to enable the driver driving mode, the driving right allocation coefficient λ is set to 1; when the driver wants to enable the automatic driving mode, the driving right allocation coefficient λ is set to 0; when the driver wants to enable the human-machine co-driving driving mode, the driving right allocation coefficient λ∈(0,1). The driver's state is identified through the driver's state identification model, and then the driving right allocation coefficient is determined based on the human-machine driving right allocation model of the driver's state. The vehicle's lateral and longitudinal control signals are finally output to control the drive execution unit, the braking execution unit and the steering execution unit to realize the acceleration, deceleration and steering control of the vehicle, completing human-machine co-driving.

[0010] Furthermore, the vehicle camera collects RGB image information; the millimeter-wave radar collects target-level data, including the distance and speed between the radar and the target object; the lidar collects 3D point cloud information; the ultrasonic radar collects raw data; and the inertial navigation system on-board sensor collects vehicle position and speed information.

[0011] Furthermore, the specific method of the automatic driving decision system outputting the lateral and longitudinal control signals a2 is as follows:

[0012] S11, aligning RGB image information of different frequencies, 3D point cloud information, raw ADC data, and ultrasonic radar raw data through interpolation or sliding window, and unifying the data into the vehicle coordinate system;

[0013] S12. Obtain feature representations of different modalities , , For the feature dimensions of different modalities, a cross-modal attention mechanism is used to fuse the image features and point cloud features. First, the features of each modality are linearly transformed to obtain the query vector , key vector Sum vector , the calculation formula is as follows:

[0014] ,

[0015] ,

[0016] ,

[0017] Where, , and is the weight matrix to be learned;

[0018] Calculate the attention score between each modality and all other modalities, i The first mode and the j The calculation formula of the attention score of the modality is:

[0019] ,

[0020] Where, is the scaling factor; For the j The key vector of the mode; is the key vector of different sensor modalities; l ∈{ L,M,U,C,I}; According to the calculated attention score, the values are weighted and summed to get the i The fusion features of the modalities are combined to obtain the fusion representation result , the calculation formula is:

[0021] ,

[0022] Where, For the j The value vector of the modes;

[0023] Finally, the fusion features of all modalities are concatenated or summed to obtain the final fusion feature , the calculation formula is:

[0024] ,

[0025] In the formula, “||” represents the vector concatenation operation; The feature representation result for the lidar; The characteristic representation results of millimeter wave radar; The characteristic representation result of ultrasonic radar; Feature representation results for vehicle cameras; Characterization results for inertial navigation system onboard sensors;

[0026] S13. Capture the temporal dependencies of dynamic scenes through the Transformer model, and use the imitation learning method to generate continuous lateral and longitudinal control signals a2 based on MLP.

[0027] Furthermore, the driver state recognition model is constructed as follows:

[0028] S21. Input the model into the feature vector Expressed as:

[0029] ,

[0030] Where, ( , ) is the position of the vehicle; ( , ) is the horizontal and vertical speed; ( , ) is the horizontal and vertical acceleration information; is the minimum collision time with the surrounding vehicles; is the lane departure degree;

[0031] S22, normalize each feature dimension to preserve temporal continuity;

[0032] S23. Define the driver's driving state as a label. Calculate the fatigue index P based on the percentage of blink duration t1 and eye closure duration t2 within a certain time interval, where P = t1 / t2. Based on the fatigue index P, classify the driving state labels into four categories: normal, distracted, fatigued, and dangerous. Use one-hot encoding. When P ≥ 0.8, the driving state is defined as dangerous; when 0.6 < P < 0.8, the driving state is defined as fatigued; when 0.4 < P ≤ 0.6, the driving state is defined as distracted; and when 0 < P ≤ 0.4, the driving state is defined as normal.

[0033] S24, using the long short-term memory network LSTM as the backbone network, the output layer is activated by the softmax function, and the output model predicts the probability distribution as shown below:

[0034] ,

[0035] Where, Predict probability distributions for the model;

[0036] S25. Calculate the driver state identification model loss function, as shown in the formula:

[0037] ,

[0038] Where N is the number of samples; C=4 is the number of driving state categories; is the true value of the i-th sample under different states; is the predicted value of the i-th sample under different states; the historical 3s driving environment information data is input into the long short-term memory network to capture the temporal evolution characteristics of driving behavior and predict the driver's driving state in the next 1s.

[0039] Furthermore, the specific method of the human-machine driving rights allocation model in the steps is as follows:

[0040] The driving rights allocation calculation is as follows:

[0041] ,

[0042] Where, =[ , ], =[ , ], 、 , 、 They are the lateral acceleration and longitudinal acceleration output by the driver and the lateral acceleration and longitudinal acceleration output by the automatic driving system respectively; The driving right allocation coefficient maps the driver's driving status to a value of 0,1 as the driving right allocation coefficient.

[0043] In a second aspect, the present invention further provides a human-machine co-driving device for a driver with upper limb disabilities, which is used to implement a human-machine co-driving method for a driver with upper limb disabilities, comprising a longitudinal control device, a VCU, a steering mechanism, a lateral control device, a bracket, a rotating column, a screw, a first magnet, a first pedal, a spring base, a first return spring, a first base, a first Hall sensor, a protective shell, a second pedal, a second magnet, a second return spring, a shell, a second base, a second Hall sensor, and a thrust washer;

[0044] The driver with upper limb disabilities outputs the vehicle's longitudinal control signal through the longitudinal control device and the vehicle's lateral control signal through the left and right pedals of the lateral control device. The control signals of the longitudinal and lateral control devices are transmitted to the VCU via the CAN bus. The VCU encodes and distributes the longitudinal and lateral control signals. The lateral control signal is input to the steering mechanism, which drives the execution unit to operate, enabling the driver with upper limb disabilities to control the vehicle's lateral direction. The longitudinal control signal is input to the driving and braking execution unit to achieve longitudinal control of the vehicle.

[0045] The lateral control device includes two first pedals, one for controlling left turn and the other for controlling right turn; the longitudinal control device includes two identical devices, each with a second pedal for controlling acceleration and braking respectively;

[0046] The two first pedals are fixed to the first base via a rotating column and a bracket; one end of the first return spring is fixed to the spring base, and the other end is fixed to the first pedal; the spring base is fixed to the first base; a pair of first magnets are respectively installed at the ends of the other side of the first pedal; the first magnets are fixed to the first pedal via screws; when a driver with an upper limb disability steps on the first pedal, the first magnet on the other side of the first pedal is tilted, and the steering signal generated by the surrounding magnetic field is received by the first Hall sensor; the first Hall sensor is fixed to the first base via a protective shell arranged on the outside; the first Hall sensor is connected to the VCU via the CAN bus;

[0047] The second pedal is engaged with the rotating column on the second base; the second base is engaged with the outer shell to form a cavity; a second return spring is arranged in the cavity; one end of the second return spring is fixed at the upper end of the cavity, and the other end is fixed at the lower end of the cavity; a thrust washer is provided at the end of the thin end of the second pedal; a rotating groove is provided on the second pedal; a locking position for installing a second magnet is provided on the rotating groove. When a driver with upper limb disabilities steps on the second pedal, the second magnet is driven to rotate to generate a longitudinal control signal which is received by the second Hall sensor; the second Hall sensor is connected to the VCU via a CAN bus.

[0048] Furthermore, a limiting groove for limiting the rotation of the first pedal is provided on the rotating column.

[0049] Furthermore, the thrust washer is fixed to the second base and the housing to limit the travel of the second pedal pushed by the second return spring.

[0050] The beneficial effects of the present invention are:

[0051] 1. The human-machine co-driving method and device for upper limb disabled drivers of the present invention solves the problem of upper limb disabled drivers being unable to control the steering wheel to steer the vehicle. It can realize vehicle control in three modes: pure driver, pure automatic driving system, and human-machine co-driving system.

[0052] 2. The human-machine co-driving method and device for upper limb disabled drivers described in the present invention provides a steering assistance device for upper limb disabled drivers. The device detects left and right turn intentions by changing the position and speed of the foot pedal, and controls the wheels through a drive-by-wire system.

[0053] 3. The human-machine co-driving method and device for upper limb disabled drivers described in the present invention are based on an end-to-end autonomous driving decision algorithm based on a multimodal large model. The driver's driving status is used as the allocation coefficient of the human-machine driving rights allocation model, achieving a high degree of integration between the driver and the autonomous driving system, improving driving safety and enhancing the driver's driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of a human-machine co-driving method for a driver with upper limb disabilities according to the present invention;

[0056] Figure 2 This is a schematic structural diagram of a human-machine co-driving device for drivers with upper limb disabilities according to the present invention;

[0057] Figure 3 An exploded schematic diagram of the lateral control device of the present invention;

[0058] Figure 4 An exploded schematic diagram of the longitudinal control device of the present invention;

[0059] Figure 5 This is an example diagram of the autonomous driving decision-making method based on multimodal spatiotemporal feature fusion in the present invention.

[0060] In the picture:

[0061] 1. Longitudinal control device; 2. VCU; 3. Steering mechanism; 4. Lateral control device; 5. Bracket; 6. Rotating column; 7. Screw; 8. First magnet; 9. First pedal; 10. Spring base; 11. First return spring; 12. First base; 13. First Hall sensor; 14. Protective shell; 15. Second pedal; 16. Second magnet; 17. Second return spring; 18. Housing; 19. Second base; 20. Second Hall sensor; 21. Thrust washer; 22. Output port. DETAILED DESCRIPTION

[0062] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0063] Example 1:

[0064] See Figure 1 This embodiment provides a human-machine co-driving method for a driver with upper limb disabilities, including the following steps:

[0065] A. A driver with upper limb disability controls the longitudinal control device 1 with his right foot. The electronic sensor senses the depth of the accelerator pedal and converts the pedal depth into an electrical signal to generate the driver's acceleration and braking control signals. The driver controls the lateral control device 4 with his left foot to generate the vehicle steering control signal:

[0066] Among them, see Figure 5 The vehicle camera collects RGB image information; the millimeter-wave radar collects target-level data, including the distance and speed between the radar and the target; the lidar collects 3D point cloud information; the ultrasonic radar collects raw data; and the inertial navigation system on-board sensor collects vehicle position and speed information.

[0067] B. The vehicle's onboard sensors, including cameras, millimeter-wave radar, lidar, ultrasonic radar, and inertial navigation system, perceive the surrounding driving environment. The autonomous driving decision-making system, based on a multimodal large model, outputs lateral and longitudinal control signals a2, as follows:

[0068] S11, aligning RGB image information of different frequencies, 3D point cloud information, raw ADC data, and ultrasonic radar raw data through interpolation or sliding window, and unifying the data into the vehicle coordinate system;

[0069] S12. Obtain feature representations of different modalities , , For the feature dimensions of different modalities, a cross-modal attention mechanism is used to fuse the image features and point cloud features. First, the features of each modality are linearly transformed to obtain the query vector , key vector Sum value vector , the calculation formula is as follows:

[0070] ,

[0071] ,

[0072] ,

[0073] Where, , and is the weight matrix to be learned;

[0074] Calculate the attention score between each modality and all other modalities, i The first mode and the j The calculation formula of the attention score of the modality is:

[0075] ,

[0076] Where, for The transpose of is the scaling factor; is the key vector of the jth mode; is the key vector of different sensor modalities; l ∈{ L,M,U,C,I}, L For LiDAR, M For millimeter wave radar, U For ultrasonic radar, C For vehicle cameras, I is the inertial navigation system onboard sensor; according to the calculated attention score, the values are weighted and summed to obtain the first i The fusion features of the modalities are combined to obtain the fusion representation result , the calculation formula is:

[0077] ,

[0078] Where, For the j The value vector of the modes;

[0079] Finally, the fusion features of all modalities are concatenated or summed to obtain the final fusion feature , the calculation formula is:

[0080] ,

[0081] In the formula, “||” represents the vector concatenation operation; The feature representation result for the lidar; The characteristic representation results of millimeter wave radar; The characteristic representation result of ultrasonic radar; Feature representation results for vehicle cameras; Characterization results for inertial navigation system onboard sensors;

[0082] S13. Capture the temporal dependencies of dynamic scenes through the Transformer model, and use the imitation learning method to generate continuous lateral and longitudinal control signals a2 based on MLP.

[0083] C. The human-machine co-driving of the present invention can realize three driving modes: driver, automatic driving, and human-machine co-driving; when the driver wants to enable the driver driving mode, the driving right allocation coefficient λ is set to 1; when the driver wants to enable the automatic driving mode, the driving right allocation coefficient λ is set to 0; when the driver wants to enable the human-machine co-driving driving mode, the driving right allocation coefficient λ∈(0,1). The driver's state is identified through the driver's state identification model, and then the driving right allocation coefficient is determined based on the human-machine driving right allocation model of the driver's state. The vehicle lateral and longitudinal control signals are finally output to control the drive execution unit, the braking execution unit and the steering execution unit to realize acceleration, deceleration and steering control of the vehicle, thereby completing human-machine co-driving.

[0084] The driver status recognition model is constructed as follows:

[0085] S21. Input the model into the feature vector Expressed as:

[0086] ,

[0087] Where, ( , ) is the position of the vehicle; ( , ) is the horizontal and vertical speed; ( , ) is the horizontal and vertical acceleration information; is the minimum collision time with the surrounding vehicles; is the lane departure degree;

[0088] S22, normalize each feature dimension to preserve temporal continuity;

[0089] S23. Define the driver's driving state as a label. Calculate the fatigue index P based on the percentage of blink duration t1 and eye closure duration t2 within a certain time interval, where P = t1 / t2. Based on the fatigue index P, classify the driving state labels into four categories: normal, distracted, fatigued, and dangerous. Use one-hot encoding. When P ≥ 0.8, the driving state is defined as dangerous; when 0.6 < P < 0.8, the driving state is defined as fatigued; when 0.4 < P ≤ 0.6, the driving state is defined as distracted; and when 0 < P ≤ 0.4, the driving state is defined as normal.

[0090] S24, using the long short-term memory network LSTM as the backbone network, the output layer is activated by the softmax function, and the output model predicts the probability distribution as shown below:

[0091] ,

[0092] Where, Predict probability distributions for the model;

[0093] S25. Calculate the driver state identification model loss function as follows:

[0094] ,

[0095] Where N is the number of samples; C=4 is the number of driving state categories; is the true value of the i-th sample under different states; is the predicted value of the i-th sample under different states; the historical 3s driving environment information data is input into the long short-term memory network to capture the temporal evolution characteristics of driving behavior and predict the driver's driving state in the next 1s.

[0096] The specific method of the human-machine driving rights allocation model is as follows:

[0097] The driving rights allocation calculation is as follows:

[0098] ,

[0099] Where, =[ , ], =[ , ], 、 , 、 They are the lateral acceleration and longitudinal acceleration output by the driver and the lateral acceleration and longitudinal acceleration output by the automatic driving system respectively; The driving right allocation coefficient maps the driver's driving status to a value of 0,1 as the driving right allocation coefficient.

[0100] Example 2:

[0101] See Figure 2 、 Figure 3 and Figure 4 This embodiment provides a human-machine co-driving device for upper limb disabled drivers, which is used to implement the human-machine co-driving method for upper limb disabled drivers described in Example 1, including a longitudinal control device 1, a VCU 2, a steering mechanism 3, a lateral control device 4, a bracket 5, a rotating column 6, a screw 7, a first magnetic steel 8, a first pedal 9, a spring base 10, a first return spring 11, a first base 12, a first Hall sensor 13, a protective shell 14, a second pedal 15, a second magnetic steel 16, a second return spring 17, a shell 18, a second base 19, a second Hall sensor 20 and a thrust washer 21.

[0102] A driver with an upper limb disability outputs the vehicle's longitudinal control signals by stepping on the two second pedals 15 of the longitudinal control device 1 and the left and right first pedals 9 of the lateral control device 4. These two control signals are transmitted via the CAN bus to the VCU 2, which encodes and distributes the longitudinal and lateral control signals. The VCU 2 encodes the processed signals into the CAN bus communication protocol format, with each signal having a unique identifier and data format. The longitudinal signal is encoded as a frame containing information such as vehicle speed and acceleration request. The lateral signal is encoded as a frame containing information such as steering angle and steering speed. The lateral control signal is then input into the steering mechanism 3, which decodes it and obtains the lateral control signal. This then drives the steering gear to rotate the steering rack, with the electric power steering motor providing assist, enabling the driver with an upper limb disability to control the vehicle's lateral direction. The longitudinal control signal is then encoded by the VCU 2 and transmitted via the CAN bus to the drive and brake motors, which decode it and obtain the longitudinal control signal. This controls the operation of the drive and brake actuator units to achieve longitudinal control of the vehicle.

[0103] See Figure 3 , Figure 3 This is an exploded view of the lateral control pedal for drivers with upper limb disabilities. It features two first pedals 9 that output the vehicle's lateral control signals. These two first pedals 9 output left and right turn control signals, respectively. They are secured to a first base 12 via a rotating column 6 and a bracket 5. A stopper slot between the rotating column 6 and the pedals also restricts the movement of the two first pedals 9, ensuring their rotation in the longitudinal plane. A first return spring 11 is installed beneath each first pedal 9. This return spring 11 is secured between the spring base 10 and the first pedal 9, ensuring that after the driver with upper limb disabilities stops applying pressure to the first pedal 9, the first pedal 9 rotates back to its contact with the stopper slot, returning the first pedal 9 to its original position. To generate and output the vehicle's lateral control signals, a pair of first magnets 8 are mounted on the opposite ends of each first pedal 9. These magnets 8 are secured to the first pedal 9 via screws 7. When a driver with upper limb disabilities depresses a first pedal 9, the first magnet 8 on the other side of the first pedal 9 tilts upward, causing a change in the surrounding magnetic field, generating a turn signal. This change in magnetic field is detected by a first Hall effect sensor 13. The first Hall sensor 13 is composed of a brush and a circuit board. The protective shell 14 surrounding the first Hall sensor 13 fixes and protects it. When the brush detects changes in the surrounding magnetic field, it generates a corresponding signal, transmits the received signal to the circuit board, and then transmits it to the output port 22. Finally, it is transmitted to the VCU2 via the CAN bus for centralized processing.

[0104] See Figure 4 , Figure 4This is an exploded view of the longitudinal control device. A driver with upper limb disabilities applies a certain amount of pressure to the second pedal 15, causing it to deflect by a certain angle, thereby generating a longitudinal control signal. The second pedal 15 is mounted on a second base 19. The rotating column on the second base 19 engages with the second pedal 15 to achieve the rotation of the second pedal 15 in the longitudinal plane. At the same time, the second base 19 engages with the outer shell 18 to form a cavity, and a second return spring 17 is provided in the cavity. When the driver with upper limb disabilities stops applying pressure to the second pedal 15, the second return spring 17 can ensure that the second pedal 15 is pushed back to its initial position. In addition, a thrust washer 21 is provided at the end of the thin end of the second pedal 15. It is fixed to the second base 19 and the outer shell 18 and is used to limit the stroke of the second pedal 15 pushed by the second return spring 17. To generate and output drive and brake signals, a latch for a second magnet 16 is located near the rotation slot of the second pedal 15. When a driver with an upper limb disability depresses the second pedal 15, it rotates the second magnet 16, causing a change in the surrounding magnetic field and generating a longitudinal control signal. A second Hall effect sensor 20, located near the second magnet 16, senses these changes in the surrounding magnetic field. It receives the changing magnetic field signal through a brush head near the second magnet 16 and transmits it to the circuit board. The circuit converts the magnetic field signal into a longitudinal control signal and transmits it to output port 22. Output port 22 then outputs the longitudinal control signal to VCU2, which encodes and distributes it.

[0105] The VCU2 has a built-in program that can determine whether the driver is controlling the execution unit alone, automatically driving, or sharing driving with the machine through the driving rights allocation coefficient λ.

[0106] In summary, the present invention solves the problem that a driver with an upper limb disability cannot steer the vehicle by controlling the steering wheel with both hands.

[0107] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A human-machine co-driving method for drivers with upper limb disabilities, characterized in that: The following steps are involved: The driver with upper limb disability controls the longitudinal control device (1) with the right foot, and the electronic sensor senses the depth of the pedal depression, converts the pedal depth into an electrical signal, and generates the driver's acceleration and braking control signal; and controls the lateral control device (4) with the left foot, and generates the vehicle steering control signal; The vehicle's onboard sensors, including cameras, millimeter-wave radars, lidars, ultrasonic radars, and inertial navigation systems, perceive the surrounding driving environment. The autonomous driving decision-making system, based on a multimodal large model, outputs lateral and longitudinal control signals a2. Human-machine co-driving realizes three driving modes: driver, automatic driving and human-machine co-driving; when the driver wants to enable the driver driving mode, the driving right allocation coefficient λ is set to 1; when the driver wants to enable the automatic driving mode, the driving right allocation coefficient λ is set to 0; When the driver wants to enable the human-machine co-driving mode, the driver's state is identified through the driver's state identification model. The human-machine driving right allocation model based on the driver's state determines the driving right allocation coefficient. The vehicle's lateral and longitudinal control signals are finally output to control the drive execution unit, brake execution unit and steering execution unit to achieve acceleration, deceleration and steering control of the vehicle, thus completing human-machine co-driving. The specific method of the automatic driving decision system outputting the lateral and longitudinal control signals a2 is as follows: S11, aligning RGB image information of different frequencies, 3D point cloud information, raw ADC data, and ultrasonic radar raw data through interpolation or sliding window, and unifying the data into the vehicle coordinate system; S12. Obtain feature representations f of different modalities i , f i ∈R di , R di For the feature dimensions of different modalities, a cross-modal attention mechanism is used to fuse the image features and point cloud features. First, the features of each modality are linearly transformed to obtain the query vector q i , key vector k i Sum value vector v i , the calculation formula is as follows: Where, and is the weight matrix to be learned; Calculate the attention score between each modality and all other modalities. The attention score calculation formula between the i-th modality and the j-th modality is: Where, q i The transpose of d k is the scaling factor; k j is the key vector of the jth mode; k l is the key vector of different sensor modalities; l∈{L,M,U,C,I}, L is the lidar, M is the millimeter wave radar, U is the ultrasonic radar, C is the vehicle camera, and I is the inertial navigation system on-board sensor; according to the calculated attention score, the values are weighted summed to obtain the fusion feature of the i-th modality to obtain the fusion representation result z i , the calculation formula is: Where, v j is the value vector of the jth mode; Finally, the fusion features of all modalities are concatenated or summed to obtain the final fusion feature z. The calculation formula is: z=[z L ||with M ||with U ||with C ||with I ] In the formula, "||" represents vector concatenation operation; z L is the feature representation result of the laser radar; z M is the characteristic representation result of millimeter wave radar; z U is the characteristic representation result of ultrasonic radar; z C is the feature representation result of the vehicle camera; z I Characterization results for inertial navigation system onboard sensors; S13. Capture the temporal dependencies of dynamic scenes through the Transformer model, and use the imitation learning method to generate continuous lateral and longitudinal control signals a2 based on MLP.

2. The human-machine co-driving method for upper limb disabled drivers according to claim 1, characterized in that: The vehicle camera collects RGB image information; the millimeter-wave radar collects target-level data, including the distance and speed between the radar and the target object; the lidar collects 3D point cloud information; the ultrasonic radar collects raw data; and the inertial navigation system on-board sensor collects vehicle position and speed information.

3. The human-machine co-driving method for upper limb disabled drivers according to claim 1, characterized in that: The method for constructing the driver state identification model is as follows: S21. Input the model into the feature vector x t Expressed as: x t =[p x ,p y ,v x ,v y ,a x ,a y ,ttc min ,d l ], Where, (p x ,p y ) is the position of the vehicle; (v x ,v y ) is the transverse and longitudinal speed; (a x ,a y ) is the horizontal and vertical acceleration information; ttc min is the minimum collision time with the surrounding vehicles; d l is the lane departure degree; S22, normalize each feature dimension to preserve temporal continuity; S23. Define the driver's driving state as a label. Calculate the fatigue index P based on the percentage of blink duration t1 and eye closure duration t2 within a certain time interval, where P = t1 / t2. Classify the driving state labels into four categories: normal, distracted, fatigued, and dangerous based on the fatigue index P, using one-hot encoding. When P ≥ 0.8, the driving state is defined as dangerous; when 0.6 < P < 0.8, the driving state is defined as fatigue; When 0.4<P≤0.6, the driving state is defined as distracted; When 0<P≤0.4, the driving state is defined as normal; S24, using the long short-term memory network LSTM as the backbone network, the output layer is activated by the softmax function, and the output model predicts the probability distribution as shown below: Where, Predict probability distributions for the model; S25. Calculate the driver state identification model loss function as follows: Where N is the number of samples; C = 4, is the number of driving state categories; y i,c is the true value of the i-th sample under different states; is the predicted value of the i-th sample under different states; the historical 3s driving environment information data is input into the long short-term memory network to capture the temporal evolution characteristics of driving behavior and predict the driver's driving state in the next 1s.

4. The human-machine co-driving method for upper limb disabled drivers according to claim 1, characterized in that: The specific method of the human-machine driving rights allocation model is as follows: The driving rights allocation calculation is as follows: a f =a1λ+(1-λ)a2, Where a1=[a m ,a n ],a2=[a p ,a q ], a m 、a n , a p 、a q are the lateral acceleration and longitudinal acceleration output by the driver and the lateral acceleration and longitudinal acceleration output by the automatic driving system respectively; λ is the driving right allocation coefficient, which maps the driver's driving state to a value of 0,1 as the driving right allocation coefficient.

5. A human-machine co-driving device for upper limb disabled drivers, used to implement the human-machine co-driving method for upper limb disabled drivers as described in any one of claims 1 to 4, characterized in that: The invention comprises a longitudinal control device (1), a VCU (2), a steering mechanism (3), a lateral control device (4), a bracket (5), a rotating column (6), a screw (7), a first magnetic steel (8), a first pedal (9), a spring base (10), a first return spring (11), a first base (12), a first Hall sensor (13), a protective shell (14), a second pedal (15), a second magnetic steel (16), a second return spring (17), a shell (18), a second base (19), a second Hall sensor (20) and a thrust washer (21); The driver with upper limb disability outputs a longitudinal control signal of the vehicle by stepping on a longitudinal control device (1), and outputs a lateral control signal of the vehicle by stepping on the left and right pedals of a lateral control device (4); the control signals of the longitudinal control device (1) and the lateral control device (4) are transmitted to a VCU (2) via a CAN bus; the VCU (2) encodes and distributes the longitudinal control signal and the lateral control signal; the lateral control signal is input to a steering mechanism (3), and the steering mechanism (3) drives an execution unit to operate, thereby realizing lateral control of the vehicle by the driver with upper limb disability; the longitudinal control signal is input to a driving and braking execution unit, thereby realizing longitudinal control of the vehicle; The lateral control device (4) includes two first pedals (9), one of which is used to control left turning and the other is used to control right turning; the longitudinal control device (1) includes two identical devices, each of which has a second pedal (15) for controlling acceleration and braking respectively; Two first pedals (9) are fixed to the first base (12) through a rotating column (6) and a bracket (5); one end of the first return spring (11) is fixed to the spring base (10), and the other end is fixed to the first pedal (9); the spring base (10) is fixed to the first base (12); a pair of first magnets (8) are respectively installed at the ends of the other side of the first pedal (9); the first magnets (8) are fixed to the first pedal (9) through screws (7); when the upper limb disabled driver steps on the first pedal (9), the first magnet (8) on the other side of the first pedal (9) is tilted, and the steering signal generated by the surrounding magnetic field is received by the first Hall sensor (13); the first Hall sensor (13) is fixed to the first base (12) through a protective shell (14) arranged on the outside; the first Hall sensor (13) is connected to the VCU (2) through a CAN bus; The second pedal (15) is engaged with the rotating column on the second base (19); the second base (19) is engaged with the housing (18) to form a cavity; a second return spring (17) is arranged in the cavity; one end of the second return spring (17) is fixed to the upper end of the cavity, and the other end is fixed to the lower end of the cavity; a thrust washer (21) is arranged at the end of the thin end of the second pedal (15); a rotating groove is provided on the second pedal (15); a locking position for installing a second magnet (16) is provided on the rotating groove, and when the upper limb disabled driver steps on the second pedal (15), the second magnet (16) is driven to rotate to generate a longitudinal control signal which is received by the second Hall sensor (20); the second Hall sensor (20) is connected to the VCU (2) via a CAN bus.

6. The human-machine co-driving device for upper limb disabled drivers according to claim 5, characterized in that: The rotating column (6) is provided with a limiting groove for limiting the rotation of the first pedal (9).

7. The human-machine co-driving device for upper limb disabled drivers according to claim 5, characterized in that: The thrust washer (21) is fixed on the second base (19) and the housing (18) and is used to limit the travel of the second pedal (15) pushed by the second return spring (17).

Images