Man-machine co-driving control right adaptive switching method based on in-vehicle and out-vehicle collaborative perception

Abstract

The application provides a man-machine co-driving control right adaptive switching method based on in-vehicle and out-vehicle collaborative perception. The method comprises the following steps: S11, a driver controls a man-machine co-driving vehicle; S12, an out-vehicle perception module is used to judge an external environment driving risk level, and an in-vehicle perception module is used to identify a driver distraction danger level; S13, a driving state of the vehicle is determined by coupling the driving risk level judgment result and the driver distraction danger level identification result; S14, according to the driving state determination result of the vehicle, whether the conditions for the automatic driving system to take over the man-machine co-driving vehicle are further analyzed; S15, if the conditions are met, the driver and the automatic driving system are adaptively assigned the control weight of the man-machine co-driving vehicle, otherwise the driver continues to control the man-machine co-driving vehicle; S16, according to the calculated control weight, the control right is gradually switched to the automatic driving system, and finally the automatic driving system controls the man-machine co-driving vehicle. The application adaptively adjusts the control right between the man and the machine by fusing the perceived surrounding environment information and the driver state information in the vehicle, so as to improve the man-machine cooperation rate, reduce the man-machine conflict, and improve the driving safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving of automobiles and the technical field of human-machine co-driving, and in particular to a human-machine co-driving automobile control right adaptive switching method based on in-vehicle and out-vehicle cooperative perception. BACKGROUND

[0002] Automatic driving technology is an effective way to reduce the incidence of traffic accidents and reduce the workload of drivers. In recent years, it has attracted a large number of enterprises and scholars to study. Considering the complexity of urban road traffic environment, before the automatic driving technology is fully mature, there will be a long period of time for drivers to participate in the control process of the automatic driving automobile, that is, the human-machine co-driving mode between manual driving and fully automatic driving will exist for a long time.

[0003] Human-machine co-driving technology is a new mode in which a human driver and an automatic driving system jointly control a vehicle through a shared control system, aiming to solve the driving control right conversion problem in the L3 (highly automatic driving) level. When the vehicle is in a dangerous scene, the driving subjects (drivers and automatic driving systems) in the shared control can control the vehicle using their respective advantages to improve driving safety. Currently, the allocation of control rights between drivers and automatic driving systems has become one of the focuses of attention in the field of automatic driving technology and the field of human-machine co-driving technology. For example, Chinese invention patent (CN201910700881) discloses a driving right switching system considering the state of a driver in a human-machine co-driving environment, which judges whether the vehicle and the driver are in an unsafe state by fusing vehicle state parameters such as vehicle speed, steering wheel angle and GPS positioning information, and driver physiological information, and hard switches the driving control right according to the safety state judgment result. Chinese invention patent (CN202110848303) establishes a human-machine co-driving model based on method bias by fitting the data of driving state and vehicle acceleration, and indirectly adjusts the dangerous operation of the driver, but the model only studies the control right switching according to the response of the driver to the risk of the vehicle surrounding environment. These control right switching methods rarely consider the coupling relationship between vehicle driving risk and driver distraction state, and it is difficult to ensure the consistency of the driver and the automatic system.

[0004] Therefore, in the human-machine co-driving state, how to fuse the in-vehicle and out-vehicle perception information to adaptively allocate the control right of the human-machine co-driving automobile is a key problem that needs to be solved. SUMMARY

[0005] In view of the deficiencies of existing research, the present application provides a human-machine co-driving automobile control right adaptive switching method based on in-vehicle and out-vehicle cooperative perception, which adaptively allocates the driving control right of the driver and the automatic driving system by real-time detection of the surrounding driving environment and identification of the driver's distraction state, and enhances the control ability of the automatic driving system to the vehicle during and after the conversion.

[0006] The application specifically adopts the following technical solutions to achieve the above-mentioned purpose.

[0007] A man-machine co-driving automobile control right adaptive switching method based on in-vehicle and out-vehicle collaborative perception, the method comprising:

[0008] S11, controlling the man-machine co-driving automobile by the driver;

[0009] S12, judging the external environment driving risk level by using the out-vehicle perception module, and identifying the driver distraction danger level by using the in-vehicle perception module. The external environment driving risk level is judged by using the out-vehicle perception module, including data collection, calculation of driving risk field force, calculation of risk index, and judgment of driving risk level; wherein the driving risk level is no risk, low risk and high risk. The driver distraction danger level is identified by using the in-vehicle perception module, including classifying common driver distraction behaviors and dividing the danger levels, and constructing a long short-term memory neural network model with attention mechanism for driver distraction behavior identification; wherein the corresponding relationship between the driver distraction behavior and the danger level is that the driver safe driving is no distraction, the driver stretching his hand behind or chatting with passengers is low distraction, the driver making up or adjusting the vehicle-mounted equipment is medium distraction, and the driver using the mobile phone to send messages or make calls is high distraction.

[0010] S13, judging the self-vehicle driving state. The self-vehicle driving state is judged by coupling the driving risk level and the driver distraction danger level, including no danger, danger and very dangerous driving states.

[0011] S14, further analyzing whether the conditions for the automatic driving system to take over the man-machine co-driving automobile are met according to the self-vehicle driving state judgment result, that is, when the self-vehicle driving state is in the dangerous or very dangerous driving state, step S15 is executed, otherwise step S11 is executed.

[0012] S15, adaptively allocating the control weight of the man-machine co-driving automobile between the driver and the automatic driving system according to the driving risk index, the driving risk level and the driver distraction danger level, and smoothly transferring the driving control right to the automatic driving system when necessary.

[0013] S16, controlling the man-machine co-driving automobile by the automatic driving system. After the automatic driving system completes the takeover, it continues to perform the driving task until passing through the dangerous road section, and when the man-machine co-driving automobile successfully passes through the dangerous road section, the driver can choose to continue to control the vehicle or choose to continue to drive by the automatic driving system.

[0014] Further, the external environment driving risk level judged by using the out-vehicle perception module further comprises:

[0015] Data acquisition. Real-time acquisition of vehicle data using CAN bus and vehicle-mounted laser radar sensor, including the speed, acceleration and headway of the ego vehicle and other surrounding vehicles, etc.

[0016] Calculate the field force of the risk field. According to the lane line information detected by the vehicle-mounted camera, a driving risk field model of the lane line potential field related to the lane is established;

[0017] Calculate the driving risk index. According to the risk field model of the surrounding driving environment, calculate the driving risk index;

[0018] Judge the driving risk level. Compare the risk index at each time with different driving risk thresholds to judge the driving risk level of the surrounding environment at the current time.

[0019] Further, the use of the external perception module to judge the driving risk level of the external environment specifically includes the following steps:

[0020] S211, data acquisition

[0021] Real-time acquisition of vehicle data using CAN bus and vehicle-mounted laser radar sensor, including the speed, acceleration and headway of the ego vehicle and other surrounding vehicles, etc.

[0022] S212, calculate the field force of the risk field

[0023] According to the lane line information detected by the vehicle-mounted camera, the lane line direction is taken as the Y axis of the road coordinate system, and the lane line vertical direction is taken as the X axis of the road coordinate system, to establish a driving risk field model of the potential field related to the lane:

[0024] E T +E L +E B +E V (1)

[0025] In formula (1), E T is the total field strength of the driving risk field, E L is the lane line potential field strength, E B is the road boundary potential field strength represented by E V is the vehicle potential field strength;

[0026] In formula (1), the lane line potential field strength composed of the lane line potential field and the double yellow line potential field in the road environment is calculated by the following formula (2):

[0027]

[0028] In formula (2), A i(i = 1, 2) represent the field strength coefficients of different types of lane line potential fields, x represents the horizontal coordinate value of the position of the ego vehicle in the road coordinate system, x l,j represents the position coordinate of the jthlane line along the X-axis direction, represents the distance from the ego vehicle to the jthlane line, and p represents the rate of change of the lane line potential field with the speed of the ego vehicle approaching or moving away from the lane line;

[0029] In formula (1), the field strength of the road boundary potential field generated at the left and right boundaries of the road is calculated using the following formula (3):

[0030]

[0031] In formula (3), x b,z represents the position coordinate of the zthroad boundary line along the X-axis direction, represents the distance from the ego vehicle to the zthroad boundary line, and p represents the rate of change of the lane line potential field with the speed of the ego vehicle approaching or moving away from the lane line;

[0032] In formula (1), the field strength of the vehicle potential field containing motion state information is calculated using the following formula (4):

[0033]

[0034] In formula (4), M represents the equivalent mass of the ego vehicle, m represents the actual mass of the ego vehicle, v represents the driving speed of the ego vehicle at the current time, d' represents the safety distance from the spatial coordinates (x0, y0) of the center of mass of a certain surrounding vehicle to the spatial coordinates (x, y) of the center of mass of the ego vehicle, τ represents the critical threshold of the safety distance between the surrounding vehicle and the ego vehicle, θ represents the angle formed by the line connecting the center of mass of a certain surrounding vehicle and the center of mass of the ego vehicle and the direction of vehicle motion, and a represents the acceleration of the ego vehicle at the current time;

[0035] S213, calculating the driving risk index

[0036] According to formula (1), a risk field model of the surrounding driving environment is established, and the driving risk index RI at time t is calculated t :

[0037]

[0038] In formula (5), F represents the field force of the risk field on the ego vehicle, and F* represents the standard risk index;

[0039] S214, judging the driving risk level

[0040] Let ω1 and ω2 be the low-risk and high-risk driving thresholds, respectively, and the calculated driving risk index RI t is compared with different driving risk thresholds to judge the driving risk level R t of the surrounding environment at the current time, and the specific process is:

[0041] (1) If RI t <ω1, then R t =R0, which means the current driving risk level is R0, indicating that the surrounding driving environment is risk-free, and the risk intensity value of this level is

[0042] (2) If ω1 t <ω2, then R t =R1, which means the current driving risk level is R1, indicating that the surrounding driving environment is low-risk, and the risk intensity value of this level is

[0043] (3) If RI t >ω2, then R t =R2, which means the current driving risk level is R2, indicating that the surrounding driving environment is high-risk, and the risk intensity value of this level is

[0044] Further, the method for identifying the distraction risk level of the driver by using the in-vehicle sensing module further comprises:

[0045] Classifying common distraction behaviors of drivers. Preprocess the videos and pictures of driver distraction behaviors in the State Farm Distracted Driver Detection public dataset to obtain six common distraction behaviors, including the driver stretching his hand behind, the driver chatting with passengers, the driver making up, the driver adjusting the vehicle-mounted device, the driver sending text messages and making phone calls;

[0046] Dividing the risk level of common distraction behaviors. According to the risk degree of each type of distraction behavior, four risk levels are divided, and the specific corresponding relationship is as follows: the driver is safe driving and is marked as no distraction D0, the driver stretches his hand behind or chats with passengers and is marked as low distraction D1, the driver makes up or adjusts the vehicle-mounted device and is marked as medium distraction D2, and the driver sends text messages or makes phone calls and is marked as high distraction D3. Mark the distraction risk level of 22424 collected driving images, and the sample number of each risk level behavior is about 5600. Divide the pre-marked dataset into training set and test set according to the ratio of 9:1, wherein the training set is used to optimize the driver distraction behavior recognition model, and the test set is used to test the classification accuracy of the recognition model. The training set and the test set both contain the behaviors of no distraction, low distraction, medium distraction and high distraction of the driver;

[0047] The long short-term memory neural network model with attention mechanism is constructed for driver distraction behavior recognition.

[0048] Further, the determination of the driving state of the ego vehicle further comprises:

[0049] The driving risk level is coupled with the driver distraction risk level, and the driving state of the ego vehicle at the current time is determined, including three driving states of no danger, danger and very dangerous.

[0050] Further, the adaptive allocation of the control weight of the driver and the automatic driving system for the man-machine co-driving vehicle further comprises:

[0051] According to the driving risk index, the driving risk level and the driver distraction risk level, the corresponding driving control weight of the driver and the automatic driving system is adaptively calculated, and the driving control weight is smoothly transferred to the automatic driving system when necessary.

[0052] The present application has the following advantages due to the above technical solutions:

[0053] 1. The driving risk level is determined by using the driving risk field, which can well quantify and predict the driving collision risk;

[0054] 2. The long short-term memory network with attention mechanism is used to identify the driver distraction risk level, which can capture the time characteristics of the driver distraction behavior in the continuous frame sequence, and better identify the distraction behavior of different risk levels;

[0055] 3. By fusing the driving risk level determination result and the driver distraction risk level identification result, the current driving state of the ego vehicle is determined, and the corresponding driving control weight of the driver and the automatic driving system is adaptively allocated, which can effectively reduce the man-machine conflict and enhance the control ability of the automatic driving system to the vehicle.

[0056] The present application proposes a kind of man-machine co-driving vehicle control weight adaptive switching method based on inside and outside vehicle collaborative perception, by fusing the surrounding environment information and the driver state information in vehicle perceived adaptively adjusts the control weight between man and machine, to improve man-machine cooperation rate and travel safety. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The figure is a working process schematic diagram of a human-machine co-driving automobile control right adaptive switching method based on in-vehicle and out-vehicle collaborative perception according to an embodiment of the present application;

[0058] Figure 2 The figure is a working process diagram of judging an external environment driving risk level by using an out-vehicle perception module according to an embodiment of the present application;

[0059] Figure 3 The figure is a three-dimensional schematic diagram of a lane line potential field strength in the field force of the risk field;

[0060] Figure 4 The figure is a three-dimensional schematic diagram of a road boundary potential field strength in the field force of the risk field;

[0061] Figure 5 The figure is a three-dimensional schematic diagram of a vehicle potential field strength in the field force of the risk field;

[0062] Figure 6 The figure is a working process diagram of identifying a driver distraction risk level by using an in-vehicle perception module according to an embodiment of the present application;

[0063] Figure 7 The figure is a working process diagram of a control right adaptive switching module according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to clearly show the technical solutions of the embodiments of the present application, the present application will be introduced below in combination with specific embodiments and drawings.

[0065] Figure 1 The figure is a working process diagram of a human-machine co-driving automobile control right adaptive switching method based on in-vehicle and out-vehicle collaborative perception according to an embodiment of the present application, which includes the following specific steps:

[0066] S11, the driver controls the human-machine co-driving automobile. In this stage, the driver controls the human-machine co-driving automobile throughout, and the automatic driving system is in a waiting state.

[0067] S12, judging the driving risk level of the external environment by using the outside perception module, and identifying the distraction risk level of the driver by using the inside perception module. The driving risk level of the external environment is judged by using the outside perception module, including data collection, calculation of the field force of the driving risk field, calculation of the risk index, and judgment of the driving risk level. The driving risk level is no risk, low risk, and high risk. The distraction risk level of the driver is identified by using the inside perception module, including classification of common distraction behaviors of the driver and division of the risk level, and construction of a long short-term memory neural network model with an attention mechanism for identification of the distraction behaviors of the driver. The corresponding relationship between the distraction behaviors of the driver and the risk level is that safe driving of the driver is no distraction, the driver stretching his hand to the back or chatting with passengers is low distraction, the driver making up or adjusting a vehicle-mounted device is medium distraction, and the driver using a mobile phone to send a message or make a call is high distraction.

[0068] S13, judging the driving state of the ego vehicle. The driving risk level of the external environment and the distraction risk level of the driver obtained in S12 are coupled to judge the driving state of the ego vehicle at the current time, including no danger, danger, and very dangerous driving states.

[0069] S14, further analyzing whether the condition for the automatic driving system to take over the ego vehicle is met according to the judgment result of the driving state of the ego vehicle, that is, when the driving state of the ego vehicle is in a dangerous or very dangerous driving state, step S15 is executed, otherwise step S11 is executed.

[0070] S15, adaptively allocating the control weight of the ego vehicle to the driver and the automatic driving system according to the driving risk index, the driving risk level, and the distraction risk level of the driver, and smoothly transferring the driving control to the automatic driving system when necessary.

[0071] S16, controlling the ego vehicle by the automatic driving system. After the automatic driving system completes the takeover, the driving task is continued until the dangerous road section is passed, and when the ego vehicle successfully passes the dangerous road section, the driver can choose to continue to control the vehicle or choose to continue to drive by the automatic driving system.

[0072] Figure 2 The working flow chart for judging the driving risk level of the external environment by using the outside perception module in the embodiment of the present application, Figure 1 can be realized by the outside perception module shown in Figure 2 , including the following specific steps:

[0073] S211, data collection. Vehicle data are collected in real time by using a CAN bus and a vehicle-mounted laser radar sensor, including the speed, acceleration, and vehicle headway of the ego vehicle and other surrounding vehicles.

[0074] S212, Calculate the field force of the risk field. Based on the lane line information detected by the vehicle camera, using the lane line direction as the Y-axis of the road coordinate system and the perpendicular direction of the lane line as the X-axis, a driving risk field model related to the lane's potential energy field is established:

[0075] E T =E L +E B +E V (1)

[0076] In equation (1), E T E represents the total field strength of the driving risk field. L For the potential energy field strength of the lane line, E B To represent the potential energy field strength at the road boundary, E V The potential energy field strength of the vehicle;

[0077] In equation (1), the potential energy field strength of the lane line in the road environment, which consists of the potential energy fields of the lane divider and the double yellow lines, is calculated using the following equation (2):

[0078]

[0079] In equation (2), A i (i = 1, 2) represent the field strength coefficients of different types of lane line potential energy fields, which determine the maximum value of the lane line field strength. Here, A1 represents the field strength coefficient of the lane dividing line potential energy field, and A2 represents the field strength coefficient of the double yellow line potential energy field. Typically, A2 > A1. x represents the abscissa value of the vehicle's position in the road coordinate system. l ,j represents the position coordinate of the j-th lane line along the X-axis. ρ represents the distance from the vehicle to the j-th lane line, and ρ represents the rate of change of the lane line potential energy field as the vehicle approaches or moves away from the lane line.

[0080] The potential energy field generated by the surrounding vehicles and the vehicle itself is as strong as Figure 3 As shown in the figure, x = 3.75m and x = 7.5m are two lane dividing lines, and x = 11.25m is a double yellow line. The figure shows that the potential energy field intensity of the lane dividing lines is much lower than that of the double yellow lines, and the potential energy field intensity is lowest in the middle of each lane, which meets the requirement that vehicles stay in the middle of the lane as much as possible.

[0081] In equation (1), the potential energy field strength of the road boundary generated at the left and right boundaries of the road is calculated using the following equation (3):

[0082]

[0083] In equation (2), x b z represents the position coordinate of the z-th road boundary line along the X-axis. η represents the distance from the vehicle to the z-th road boundary line, and η represents the road boundary field strength coefficient.

[0084] The potential energy field generated by the surrounding vehicles and the vehicle itself at the road boundary is as strong as Figure 4 As shown in the diagram, the road boundary is located at the far left of the road. The smaller the distance from the road boundary, the larger the potential energy field at the road boundary, reaching infinity at the far left of the road.

[0085] In equation (1), the potential energy field strength of the vehicle, which contains motion state information, is calculated using the following equation (4):

[0086]

[0087] In equation (4), M represents the equivalent mass of the vehicle, m represents the actual mass of the vehicle, v represents the speed of the vehicle at the current moment, d' represents the safe distance from the spatial coordinates (x0, y0) of the centroid of a certain surrounding vehicle to the spatial coordinates (x, y) of the centroid of the vehicle, τ represents the critical threshold of the safe distance between the surrounding vehicle and the vehicle, θ represents the angle formed by the line connecting the centroids of a certain surrounding vehicle and the centroid of the vehicle and the direction of vehicle motion, and a represents the acceleration of the vehicle at the current moment.

[0088] The potential energy field generated by the surrounding vehicles and the vehicle itself is as strong as Figure 5 As shown in the figure, the vehicle potential field is elliptical in distribution, indicating that the vehicle is allowed to be relatively close to the side vehicles or lane lines during driving. The vehicle potential field strength reaches its maximum value at the center of the vehicle and gradually decreases as the safe distance d' increases.

[0089] S213, Calculate the driving risk index. Based on equation (1), establish a risk field model of the surrounding driving environment and calculate the driving risk index RI at time t. t :

[0090]

[0091] In equation (5), F represents the field force of the vehicle in the risk field, and F* represents the standard risk index. In previous studies, the time difference between collisions (TTC) and the following distance (THW) have been widely considered as measures of the potential driving risk of a vehicle in dangerous scenarios. A THW of 1 second and a TTC of 4 seconds are recommended warning standards; therefore, the field force value F* experienced by the vehicle at a TTC of 4 seconds is used as the standard risk index.

[0092] S214, Determine the driving risk level. Set ω1 and ω2 as low-risk and high-risk driving thresholds, respectively, and calculate the driving risk index RI. t By comparing with different driving risk thresholds, the driving risk level R of the surrounding environment at the current moment is determined. t The specific process is as follows:

[0093] (1) If RI t <ω1, then R t =R0, which means the current driving risk level is R0, indicating that the surrounding driving environment is risk-free, and the risk intensity of this level is assigned as

[0094] (2) If ω1 t <ω2, then R t =R1, which means the current driving risk level is R1, indicating that the surrounding driving environment is low-risk, and the risk intensity of this level is assigned as

[0095] (3) If RI t >ω2, then R t =R2, which means the current driving risk level is R2, indicating that the surrounding driving environment is high-risk, and the risk intensity of this level is assigned as

[0096] Figure 6 is the work flow chart of identifying the driver distraction risk level using the in-vehicle perception module proposed in the embodiments of the present application, Figure 1 the identification of the driver distraction risk level in Figure 6 is realized by the in-vehicle perception module shown in

[0097] S221, classify common distraction behaviors of drivers. Preprocess the in-vehicle driver distraction behavior videos and pictures in the public data set State Farm Distracted Driver Detection to obtain six common distraction driving behaviors, including the driver stretching his hand behind, the driver chatting with passengers, the driver making up, the driver adjusting the vehicle-mounted device, the driver sending messages and making calls by mobile phone;

[0098] S222, divide the risk level of common distraction behaviors. According to the risk degree of each type of distraction behavior, four risk levels are divided, and the specific corresponding relationship is as follows: the driver is safe driving and is marked as no distraction D0, the driver stretches his hand behind or chats with passengers and is marked as low distraction D1, the driver makes up or adjusts the vehicle-mounted device and is marked as medium distraction D2, and the driver sends messages or makes calls by mobile phone and is marked as high distraction D3. Mark the distraction risk level of 22424 collected driving images, of which the sample number of each risk level behavior is about 5600. The pre-marked data set is divided into a training set and a test set according to a ratio of 9:1, wherein the training set is used to optimize the driver distraction behavior recognition model, and the test set is used to test the classification accuracy of the recognition model. The training set and the test set both contain the behaviors of no distraction, low distraction, medium distraction and high distraction of the driver;

[0099] S223, the long short-term memory neural network model with attention mechanism is constructed for driver distraction behavior recognition, and the pre-labeled data image is transmitted to a long short-term memory (LSTM) neural network. The LSTM network is a special recurrent neural network structure with internal cells, which provides long-term and short-term memory and helps the network to process longer data sequences. The long short-term memory neural network layer with attention mechanism is composed of three parts of input gate, forget gate and output gate. Among them, h hidden units are set, the batch size is n, the input number is d, and the input is X t ∈R n×d , the hidden state of the previous time step t-1 is H t-1 ∈R n×h , and the calculation process of the long short-term memory neural network with attention mechanism is as follows:

[0100] The first step is to use the forget gate F t to determine the information to be forgotten by the cell, and the formula is as follows:

[0101] F t =σ(X t W xf +H t-1 W hf +b f ) (6)

[0102] In formula (6), σ represents the sigmoid activation function, W xf and W hf represent the weight parameters of the forget gate, and b f represents the bias parameter of the forget gate.

[0103] The second step is to determine the content of the information saved by the cell using the input gate I t , the candidate memory element and the memory element C t , and the formula is as follows:

[0104] I t =σ(X t W xi +H t-1 W hi +b i ) (7)

[0105]

[0106]

[0107] In formula (7)-(9), W xi and W hi represent the weight parameters of the input gate, and Wxc and W hc denote the weight parameters of the candidate memory gate, b i and b c denote the bias parameters of the input gate and the candidate memory cell, respectively, C t-1 denote the memory cell at the previous time step t-1.

[0108] The third step is to use the output unit to determine the output gate O t and the hidden state H t , which is as follows:

[0109] O t = σ(X t W xo + H t-1 W ho + b o ) (10)

[0110] H t = C t ⊙ tanh(O t ) (11)

[0111] In equation (10), W xo and W ho denote the weight parameters of the output gate, b o denotes the bias parameter of the output gate.

[0112] In order to make full use of all the hidden states of the LSTM memory cells in the last layer, an attention mechanism is added to the LSTM layer to assign a trainable weight value to each hidden state, which better captures the characteristics of the driver distraction behavior. The formula is as follows:

[0113]

[0114] In equation (12), N denotes the feature vector output by the LSTM layer, ε denotes the trainable weight vector, and α denotes the distribution coefficient matrix, is the feature vector output by the attention mechanism model.

[0115] Finally, the risk level of the driver distraction behavior at time t is output through a fully connected layer with a SoftMax classifier, and the output result is D t ∈ [0, 1, 2, 3]. When Dt=0, the distraction risk level recognition result is no distraction D0, and the distraction intensity value of this level is S0=0; when Dt=1, the distraction risk level recognition result is low distraction D1, and the distraction intensity value of this level is S1=1.5; when Dt=2, the distraction risk level recognition result is medium distraction D2, and the distraction intensity value of this level is S2=2; when Dt=3, the distraction risk level recognition result is high distraction D3, and the distraction intensity value of this level is S3=2.5.

[0116] Figure 1 The determination of the self-vehicle driving state S13 includes the following specific steps:

[0117] The driving risk level obtained by coupling S12 and the driver distraction risk level are used to determine the self-vehicle driving state at the current time, which includes three driving states: no risk, risk, and very high risk. The specific process is as follows:

[0118] When the surrounding driving risk level is no risk R0 and the driver distraction risk level is no distraction D0, it means that there is no risk around the vehicle and the driver has no distraction behavior, and the driver should have full control of the driving. At this time, the self-vehicle driving state is in a no-risk state.

[0119] When the surrounding driving risk level is no risk R0 and the driver distraction risk level is low distraction D1, medium distraction D2, and high distraction D3, it means that there is no risk around the vehicle, but the driver cannot control the vehicle well due to different levels of distraction behavior, and the automatic driving system should have partial control of the driving. At this time, the self-vehicle driving state is in a risk state.

[0120] When the surrounding driving risk level is low risk R1 and the driver distraction risk level is no distraction D0, low distraction D1, medium distraction D2, and high distraction D3, it means that the surrounding driving risk is low, but the vehicle is unstable due to the driver's partial distraction behavior, and the automatic driving system should have partial control of the driving. At this time, the self-vehicle driving state is in a risk state.

[0121] When the surrounding driving risk level is high risk R2 and the driver distraction risk level is no distraction D0, low distraction D1, and medium distraction D2, it means that the surrounding driving risk is high, but the driver's attention is mostly focused on observing the current driving environment changes when performing low distraction driving tasks, and the automatic driving system should have partial control of the driving. At this time, the self-vehicle driving state is in a risk state.

[0122] When the surrounding driving risk level is high risk R2 and the driver distraction risk level is high distraction D3, it means that the surrounding driving risk is very high and the driver is affected by the distraction behavior. At this time, the self-vehicle driving state is in a very high risk state.

[0123] Figure 7 The working flowchart of the control right adaptive switching module proposed in the embodiment of the present application, Figure 1 The adaptive allocation of the control right of the human-machine co-driving vehicle by the driver and the automatic driving system can be realized by the control right adaptive switching module shown in Figure 7 , which includes the following specific steps:

[0124] S411, if the condition of the automatic driving system taking over the man-machine co-driving car is met, i.e., when the driving state of the ego car is in a dangerous or very dangerous driving state, the control weight of the driver and the automatic driving system on the man-machine co-driving car is adaptively allocated according to the driving risk index, the driving risk level and the driver distraction danger level, and the driving control is smoothly transferred to the automatic driving system when necessary, and the specific process is as follows:

[0125] Based on the ten dangerous driving states of R0D1, R0D2, R0D3, R1D0, R1D1, R1D2, R1D3, R2D0, R2D1 and R2D2 in S13, under different driving risk levels, part of the driver's attention in the car will be affected by the distraction behavior, but most of the drivers still have the ability to control part of the driving right. In order to ensure the safety of driving, the driver is informed of the impending transfer of driving control by using sound warning, and the driving control is gradually transferred to the automatic driving system at the same time. At this time, the driving control weight of the driver and the automatic driving system is:

[0126]

[0127] In formula (13), μ(t) is the control weight of the automatic driving system, and v(t) is the control weight of the driver. RI m is the maximum driving risk index calculated by a large number of dangerous driving data sets, and RI m = 2.

[0128] Based on the one very dangerous driving state of R2D3 in S13, the surrounding driving risk is very high and the driver is affected by the distraction behavior, and the driver is informed by using emergency sound warning at the same time, and the driving control is immediately transferred, and the automatic driving system performs emergency takeover. The formula is as follows:

[0129]

[0130] Finally, it should be noted that the above is a specific explanation and description of the specific embodiments of the present application, which only embodies the method and core design concept of the present application, but is not limited to the above implementation method. Those skilled in the art can make equivalent modifications or improvements without violating the principles and spirits of the present application, and these changes should be within the protection scope.

Claims

1. A human-machine co-driving control right adaptive switching method based on in-vehicle and out-of-vehicle collaborative perception, characterized in that, The method comprises the following steps: S11, controlling the man-machine co-driving vehicle by the driver; S12, judging the driving risk level of the external environment by using the vehicle exterior perception module, and identifying the driver distraction danger level by using the vehicle interior perception module; S13, coupling the driving risk level judgment result and the driver distraction danger level identification result to determine the driving state of the vehicle; S14, further analyzing whether the conditions for the automatic driving system to take over the man-machine co-driving vehicle are met according to the driving state determination result of the vehicle; S15, if the conditions are met, adaptively allocating the control weight of the man-machine co-driving vehicle to the driver and the automatic driving system, otherwise continuing to control the man-machine co-driving vehicle by the driver; S16, gradually switching the control to the automatic driving system according to the calculated control weight, and finally controlling the man-machine co-driving vehicle by the automatic driving system; The driving risk level judged by the vehicle exterior perception module further comprises: data acquisition, calculation of the field force of the driving risk field, calculation of the driving risk index, and judgment of the driving risk level; wherein the driving risk level is no risk, low risk and high risk; The driving risk level judged by the vehicle exterior perception module specifically comprises the following steps: S211, data acquisition Real-time acquisition of vehicle data including the speed, acceleration and vehicle head distance of the vehicle and other surrounding vehicles by using CAN bus and vehicle-mounted laser radar sensor; S212, calculation of the field force of the risk field According to the lane line information detected by the vehicle-mounted camera, the direction of the lane line is taken as the Y axis of the road coordinate system, and the vertical direction of the lane line is taken as the X axis of the road coordinate system, to establish a driving risk field model of the potential energy field related to the lane: （1） In formula (1), E T is the total field strength of the driving risk field, E L is the field strength of the lane line potential energy field, E B is the field strength of the road boundary potential energy field represented by E V is the field strength of the vehicle potential energy field; In formula (1), the lane line potential energy field strength composed of the lane line potential energy field and the double yellow line potential energy field in the road environment is calculated by using the following formula (2): （2） In formula (2), A i (i=1, 2) represent field strength coefficients of different types of lane line potential fields, x represents a horizontal coordinate value of a position of the ego vehicle in a road coordinate system, x l,j represents a position coordinate of the jthlane line along the X-axis direction, represents a distance from the ego vehicle to the jthlane line, represents a rate of change of the lane line potential field with respect to a speed of the ego vehicle approaching or moving away from the lane line; In formula (1), the road boundary potential energy field strength generated at the left and right boundaries of the road is calculated by using the following formula (3): （3） In formula (3), x b,z represents a position coordinate of the zth road boundary line in the X-axis direction, represents a distance from the vehicle to the zth road boundary line, represents a road boundary field intensity coefficient; In formula (1), the vehicle potential energy field strength containing motion state information is calculated by using the following formula (4): （4） In formula (4), M represents an equivalent mass of the ego vehicle, m represents an actual mass of the ego vehicle, v represents a running speed of the ego vehicle at a current time, d' represents a safety distance from a spatial coordinate (x0, y0) of a mass center of a certain surrounding vehicle to a spatial coordinate (x, y) of a mass center of the ego vehicle, represents a critical threshold value of the safety distance of the surrounding vehicle and the ego vehicle, represents an included angle formed by a line connecting the mass center of the certain surrounding vehicle and the mass center of the ego vehicle and a moving direction of the vehicle, and a represents an acceleration of the ego vehicle at the current time. S213, calculation of the driving risk index A risk field model of the surrounding driving environment is established according to formula (1), and a driving risk index at time t is calculated : （5） In formula (5), F represents the field force of the risk field received by the vehicle, and F* represents the standard risk index; S214, judgment of the driving risk level Set and respectively as low-risk, high-risk driving threshold, the calculated driving risk index RI t with different driving risk threshold, to determine the current moment around the environment driving risk level R t , the specific process is: (1) If RI t t = R0, which means the current driving risk level is R0, indicating that the surrounding driving environment is risk-free, and the risk intensity value of this level is = 0.​​ (2) If < RI t , then R t = R1, which means the current driving risk level is R1, indicating that the surrounding driving environment is low risk, and the risk intensity value is = 1.5.​ (3) If RI t , then R t = R2, i.e. the current driving risk level is R2, indicating that the surrounding driving environment is high risk, and the risk intensity value of this level is = 2;​ The driver distraction danger level identified by the vehicle interior perception module further comprises: classifying common driver distraction behaviors and dividing the danger levels, and constructing a long short-term memory neural network model with attention mechanism for driver distraction behavior identification; wherein the corresponding relationship between the driver distraction behavior and the danger level is: no distraction for safe driving of the driver, low distraction for the driver stretching his hand behind or chatting with passengers, medium distraction for the driver making up or adjusting the vehicle-mounted equipment, and high distraction for the driver sending messages or making calls by using the mobile phone; The determination of the driving state of the vehicle further comprises: coupling the driving risk level and the driver distraction danger level, and determining the driving state of the vehicle at the current time, including no danger, danger and very dangerous driving state; The adaptive allocation of the driving control weight of the driver and the automatic driving system to the control of the man-machine co-driving vehicle is characterized in that: if the driving state of the ego vehicle meets the condition for the automatic driving system to take over the control of the man-machine co-driving vehicle, that is, the driving state of the ego vehicle is in a dangerous or very dangerous driving state, then the driving control weight of the driver and the automatic driving system is adaptively allocated according to the driving risk index, the driving risk level and the driver distraction risk level, and the driving control is smoothly transferred to the automatic driving system.

Images