A human-machine fusion perception method for highly automated driving

Through the human-machine fusion perception method, combined with multiple sensors and deep learning technologies, the driver's biological signals and environmental data are integrated to generate a personalized perception map, which solves the problem that the existing intelligent driving system cannot meet the perception needs of the human-machine co-driving system. , improves the security and interpretability of the system.

CN115027484BActive Publication Date: 2024-11-22JILIN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210561201.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-23
Publication Date
2024-11-22
Estimated Expiration
2042-05-23

AI Technical Summary

Technical Problem

The existing intelligent driving system can only independently perceive objects in the driving environment, and cannot meet the needs of the driver and the man-machine co-driving system. It is impossible to judge the driver's human-machine co-driving system from the perspective of the driver. Together, the needs of the perception layer.

Method used

Adopting a human-machine fusion perception method for highly autonomous driving, through data preprocessing, driving situation perception, driver's typical biological signal extraction, driver's driving intention understanding and human-machine hybrid enhanced perception map generation, combined with convolutional neural network and Hidden Mar Kov model and deep learning technology integrate data from vehicle-mounted CAN bus, cameras, lidar, millimeter-wave radar and vehicle-road collaborative equipment, identify driver head movement, eye movement signals and driving environment, and generate personalized fusion perception maps.

Benefits of technology

It realizes a comprehensive understanding of the driver's status, reduces the driver's perceived burden, improves the system safety and interpretability, is compatible with different human-machine co-driving decision planning and control systems, and has a wide range of application prospects and multiple scenarios. , feasibility under multiple operating conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115027484B_ABST
    Figure CN115027484B_ABST
Patent Text Reader

Abstract

The present invention discloses a human-machine fusion perception method for highly automated driving, which comprises the following steps: first, data preprocessing; second, driving situation perception; third, driver typical biological signal extraction; fourth, driver driving intention understanding; fifth, human-machine hybrid enhanced perception map generation. Beneficial effects: It realizes the organic combination of human drivers and intelligent driving systems; it reduces information complexity, reduces the driver's perception burden, and does not miss key information, thus ensuring the safety of the overall system; the present invention can combine advanced technical means of multiple disciplines, conforms to the development trend of automated driving technology, and has broad application prospects and feasibility in multiple scenarios and working conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a human-machine fusion perception method, and in particular to a human-machine fusion perception method for highly automated driving. Background Art

[0002] With the continuous development of automobile technology, the level of vehicle intelligence is getting higher and higher. Smart cars have become one of the main development directions of automobile technology, and have been studied and applied in all aspects of vehicles. The vehicle operation process of smart cars can be roughly summarized as: perception, decision-making and planning, and control. These three aspects have now become recognized research hotspots in the automotive industry. According to the classification standards of the Society of Automotive Engineers (SAE International), the current main technical breakthroughs are concentrated on the research related to conditional autonomous driving and highly autonomous driving at levels L3 and L4. Human-machine co-driving technology is a highly feasible and practical smart car solution that was produced under such a background.

[0003] At present, human-machine co-driving technology at home and abroad is constantly being updated and iterated. Automotive engineers around the world have proposed various solutions to solve the perception problem of human-machine co-driving. Human-machine co-driving perception technology has the following characteristics. First, maintain the human-in-the-loop characteristics. Unlike most perception technologies for highly automated driving or fully automated driving, human-machine co-driving perception technology not only requires the perception of the surrounding traffic environment, but also requires the driver to maintain at least a minimum perception of the traffic environment. This not only greatly improves the safety of the system, but also increases the interpretability and reliability of the system. Second, the perception of the car and the driver has the same status. Unlike traditional ADAS perception technology, the human-machine co-driving perception system is not just a tool to assist the driver, but an "other driver" with the same status as the driver. It cooperates with the driver to complete the overall driving perception task, thereby significantly reducing the driver's perception burden.

[0004] At present, there is no universally accepted solution for the perception scheme of human-machine co-driving due to the difficulty of the technology itself and the incompleteness of the core theoretical system. There are many problems such as insufficient accuracy, inability to ensure real-time performance, or insufficient application occasions.

[0005] Chinese patent CN201910666351.9 discloses a driving right switching and collision warning system for human-machine co-driving intelligent connected vehicles. The system perception module only perceives the surrounding traffic environment. Chinese patent CN202111100769.7 discloses an environmental perception system, method, vehicle and storage medium for intelligent driving vehicles, which integrates lidar sensors, IMU sensors, RTK positioning information and high-precision maps to perceive the driving environment around intelligent driving vehicles. Chinese patent CN 201910420661.2 discloses a method, device, equipment and storage medium for fusion processing of perception information of intelligent driving roadside equipment, which can fuse and process perception sub-information based on time calibration to achieve good perception effect. The above three patents represent the perception schemes of the current mainstream intelligent driving system, which can only perceive objects in the driving environment independently, cannot meet the concept requirement that the driver and the human-machine co-driving system of the human-machine co-driving system complement each other, and cannot judge the driver's needs for the human-machine co-driving perception layer from the driver's perspective. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the perception scheme of the existing intelligent driving system can only perform independent perception on objects in the driving environment, cannot meet the concept requirement of the human-machine co-driving system that the driver and the human-machine co-driving system complement each other, and cannot judge the driver's demand for the human-machine co-driving perception layer from the driver's perspective. Instead, a human-machine fusion perception method for highly automated driving is provided.

[0007] The human-machine fusion perception method for highly automated driving provided by the present invention comprises the following steps:

[0008] The first step is data preprocessing. The specific steps are as follows:

[0009] Step 1, data collection: collect the on-board CAN bus, camera sensor signals, laser radar millimeter-wave radar signals, and vehicle-road cooperative equipment signals carried by the intelligent vehicle, and collect the following sensor data according to a fixed sampling period: vehicle control and motion state data represented by time series generated on the on-board CAN bus, specifically including clutch, brake, and accelerator pedal control signals, transmission control signals, and steering wheel control signals; color image streams with a certain frame rate generated by each camera sensor. The camera sensors mainly include camera sensors distributed in front of the driver to identify the driver's status and camera sensors distributed around the vehicle to identify the surrounding traffic environment. They output color image streams with driver status information and color image streams with surrounding traffic environment information respectively; a list of surrounding object motion states generated by each laser radar millimeter-wave radar sensor at a certain frame rate. Millimeter-wave radars are mainly distributed around the vehicle and output a list of surrounding object motion states with surrounding traffic environment information; a list of real-time traffic status signals transmitted by the vehicle-road cooperative equipment;

[0010] Step 2: Verify and align the data: The specific steps are as follows:

[0011] Step 1: Verify the validity of sensor data: Read the values ​​of the vehicle bus signal, vehicle state measurement signal and multimodal environment sensor signal respectively and evaluate whether they are collected and generated when the sensor is working normally. Use an integer mark to represent it. A positive value indicates that the sensor is working normally, and a negative value indicates an abnormality. If the input signal is judged to be abnormal, set the integer mark to a negative number in the output signals of the first step, namely, the vehicle bus alignment signal, the vehicle state measurement alignment signal and the multimodal environment sensor alignment signal.

[0012] Step 2: Time alignment: Time alignment refers to unifying the data collected by different sensors in different time coordinate systems into the same time coordinate system based on the sensor physical hardware information, which refers to the frame rate and startup time of the camera sensor and the millimeter wave radar sensor. This makes the data of different sensors interconnected, which is conducive to subsequent data analysis.

[0013] The strategies for time alignment include but are not limited to: setting a common data buffer to store different sensor data; calculating the absolute time based on the frame rate and start time information; or setting the system relative time inside the intelligent vehicle system to set a global timestamp for all sensor data;

[0014] Step 3: Spatial alignment: Sensor spatial alignment means unifying the data collected by different sensors in their respective coordinate systems into the same coordinate system, i.e. the vehicle coordinate system, based on the sensor placement information. This allows the data from different sensors to be interconnected, which is beneficial for subsequent data analysis.

[0015] Step 4: Integrate the aligned data: After time alignment and space alignment, the sensor data is actually a traffic environment image stream with the motion status of surrounding objects and a driver status color image stream in a unified time-space coordinate system. These two parts are integrated into a structure of a unified format and sent to the second and third steps as their data input;

[0016] The second step is driving situation awareness. The specific steps are as follows:

[0017] Step 1: Head movement signal recognition: Head movement signal recognition mainly identifies the three posture angles of the driver's head posture and the spatial position of the eyes. The head movement recognition method uses a convolutional neural network. The specific structure of the convolutional neural network is as follows:

[0018] The input image is cropped into an image of 256*256*3, and processed in sequence by the first convolution layer, the first pooling layer, the second convolution layer, and the second pooling layer to obtain a 16*16*128 feature layer. Then, it passes through a neural layer composed of 128 neurons and is normalized by the softmax algorithm to obtain the coordinate positions of the facial feature points. The convolution kernel size of the first convolution layer is 3*3*3, and the convolution step size of the first convolution layer is 2; the first pooling layer adopts the maximum pooling method, and the size of the first pooling layer is 3*3, and the first pooling step size is 2. The stride of the layer is 2; the convolution kernel size of the second convolution layer is 3*3*32, and the stride of the second convolution layer is 2; the second pooling layer adopts the maximum pooling method, the size of the second pooling layer is 2*2, and the stride of the second pooling layer is 2; the aforementioned feature points include eyebrows, eye corners, corneas, pupils, nose tips, nostrils, mouths, chins and ears. According to the positions of the feature points, the active appearance model method can be used to estimate the driver's head posture, thereby obtaining the driver's head movement signal, which will be used as part of the input for understanding the driver's driving intention in the fourth step;

[0019] Step 2, eye movement signal recognition: Eye movement signal recognition mainly recognizes eye movement vector biological signals. By identifying the driver's eye movement vector, the driver's gaze behavior is modeled to obtain the driver's gaze direction. Eye movement signal recognition is similar to step 1. It mainly uses the cornea and pupil feature points output by the convolutional neural network. The position of the same feature point at the same time collected by the binocular camera is used for spatial modeling to obtain a three-dimensional eyeball model, thereby calculating the position of the eye in space and the eye movement vector. The calculated eye movement signal is passed as input to the fourth step and the driver's driving intention understanding and the fifth step of human-machine hybrid enhanced perception map generation;

[0020] The third step is to extract typical biological signals of drivers. The specific steps are as follows:

[0021] Step 1: Driving environment perception: Use the traffic environment information around the vehicle transmitted by the sensors after the preprocessing in the first step to extract the necessary traffic participants to form a typical traffic condition. The established traffic environment includes the relevant information of all traffic participants that may interact with the main vehicle in a short period of time, including the category, relative distance and relative speed; Driving environment perception also requires the recognition of lane lines and traffic sign road information. The recognition methods include but are not limited to: fitting lane lines based on computer vision image recognition algorithms, identifying traffic signs and traffic light signals; intelligent transportation system based on vehicle-road collaboration, vehicle-roadbed equipment communication to transmit lane, traffic sign and traffic light information; or traffic road information reading based on cloud-based high-precision map information; In order to realize the extraction of traffic participants, the following schemes are used for calculation:

[0022] Solution 1: Safety time algorithm: This algorithm is a forward collision warning algorithm based on the collision time between the following vehicle and the preceding vehicle. The collision time calculation formula is as follows:

[0023] TTC=D / (V r -V f )

[0024] Where D is the distance between the following vehicle and the preceding vehicle; V r Indicates the speed of the following vehicle; V f Indicates the speed of the preceding vehicle. This algorithm is used to extract traffic participants on the same lane line;

[0025] For traffic participants on different lanes, the collision time calculation formula changes to:

[0026] TTC=D / v

[0027] Where D is the distance between the expected collision position of the two vehicles and the position of the vehicle; v is the speed of the vehicle;

[0028] Setting the TTC threshold can extract the traffic participants that need to be considered in the surrounding environment;

[0029] Solution 2: Safety distance algorithm: In the actual traffic environment, there are some traffic participants that will not pose a safety risk to the vehicle, but can affect the driving of the main vehicle. There is a certain relationship between the absolute distance between these traffic objects and the main vehicle. The safety distance DS is defined. The specific formula is as follows:

[0030] DS = vm * 2

[0031] Where, vm = max{v e , v o , 10}, unit is km / h; v e is the speed of the main vehicle, in km / h; v ois the speed of surrounding objects, in km / h; DS is in m;

[0032] When DS is less than the actual distance between the host vehicle and the object, it indicates that the object is within the safe distance range and will affect the driving of the host vehicle, and should be included in the traffic environment modeling;

[0033] Step 2: Understanding driving scenarios: Driving scenarios are defined as the external environment in which the driver obtains driving-related information at the current moment. Understanding driving scenarios mainly refers to the extraction of current driving conditions based on the driving environment built in step 1. Typical driving scenarios include but are not limited to: unobstructed urban roads, congested urban roads, intersections, site parks, highways and viaducts. The extraction method uses feature comparison to divide different driving scenarios according to the values ​​of key parameters such as the density of traffic participants on the road, speed, road shape and location of traffic lights.

[0034] Step 3: Driving situation reasoning: Driving situation refers to the driving environment status and development trend that can affect the driver and the automatic driving system over a period of time. The purpose of driving situation reasoning is to comprehensively consider the current driving environment status and give the most appropriate perception map after comprehensively considering the driver's status, so as to achieve the effect of fusion enhanced perception. The input of the driving situation reasoning model is the driving situation historical data generated in step 2. The driving situation reasoning method is the Sentiment Analysis method of deep learning, which quantitatively represents the driving situation. The representation method is SA = [sa1, sa2], where sa i The value range is 1-5, sa1 indicates the traffic crisis situation, sa1=1 means there is no safety risk in the environment, and gradually increases to sa1=5, indicating that the vehicle is about to have an unavoidable traffic accident; sa2 indicates the current traffic congestion level, sa2=1 means the road is unobstructed, and gradually increases to sa2=5, indicating that the vehicle is in severe congestion;

[0035] Step 4: Understanding the driver's driving intention. The specific steps are as follows:

[0036] Step 1: Understanding the vehicle's lateral driving intention: The driver's lateral driving intention is inferred through the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle's CAN bus. The driver's lateral driving intention includes but is not limited to turning, changing lanes and cutting in; the vehicle's lateral driving intention understanding is established using a hidden Markov model. The hidden Markov model solves the problem of understanding the driver's lateral driving intention in the following three steps:

[0037] Step 1: Model data definition and collection: The problem of understanding the driver's lateral driving intention corresponds to the evaluation problem of the hidden Markov model, that is, the problem of estimating the probability value. The core of this problem is to find the probability of the driver's lateral driving intention being a certain value under a certain traffic situation and driver state;

[0038] The driver's lateral driving intention is expressed as Q, which is defined as follows:

[0039] Q = -3, -2, -1, 0, 1, 2 respectively indicate that the driver's current lateral driving intention is to turn around, turn left, change lanes to the left, keep going straight, change lanes to the right, and turn right;

[0040] The driver's head movement signal, eye movement signal, driving environment and driving operation signal transmitted by the vehicle CAN bus are input as observation variables, and after time discretization, the observation variable sequence O1, O2, ..., O t , the driver's lateral driving intention is used as the hidden state variable of the hidden Markov model, and the hidden state variable sequence Q1, Q2, ..., Q is obtained after time discretization. t ;

[0041] The parameters of the hidden Markov model are defined as λ = (π, A, B), where π represents the initial state probability of the system, A represents the state transfer matrix, B represents the emission matrix, and the initial state probability π: the probability of each state of the model appearing at the initial moment, denoted as π = (π1, π2, ..., π t ), π i Represents the probability that the initial observed variable data of the model is Oi, and the state transition probability A: the probability of the model switching between various hidden states, recorded as the matrix A = [a ij ], where a ij It means that at any time t, if the hidden state variable data is Q i , then at the next moment the hidden state variable data is Q j The probability of the emission matrix B is the probability relationship between the model observation variables and the hidden state variables, recorded as matrix B = [b jk ], where b jk It means that at any time t, if the observed variable data is O j , then at the current moment the hidden state variable data is Q k The probability of

[0042] According to the parameters defined above, experiments are conducted to collect a sufficiently large data set, and then the data set is initialized and grouped according to the K-means clustering algorithm to obtain the initial parameter λ0 of the model;

[0043] Step 2: Model training: Model training refers to repeated deep learning training based on the obtained data set and the principle of hidden Markov model to improve the accuracy of the model so that it can properly understand the driver's lateral driving intention;

[0044] The main content of model training is: using the forward-backward algorithm to calculate the observed variable data used in the current training in the collected data set train The output probability P(O train |λ), apply the EM algorithm to update the model parameters λ, specifically including calculating the new state transfer matrix A and the new emission matrix B, repeating the training by replacing the observed variable data and the corresponding hidden state variable data, and updating P(O train |λ) is monitored, and when the value converges to a very small change or no longer changes, the training ends, and the hidden Markov model parameter λ obtained after the last training is end As the final hidden Markov model parameters, a hidden Markov model for understanding the actual driver's lateral driving intention is obtained;

[0045] Link 3: Understanding the driver's lateral driving intention: After training, the hidden Markov model is used to understand the driver's lateral driving intention. The understanding process is the decoding problem solving process of the hidden Markov model. The decoding process uses the Viterbi algorithm to discretize the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus within a period of time to obtain the observed variable sequence O1, ..., O t Input the model and find the probability that the driver's lateral driving intention Q at the next moment is -3, -2, -1, 0, 1, and 2 respectively. The one with the largest probability is taken as the final output of the driver's lateral driving intention.

[0046] Step 2: Understanding the longitudinal driving intention of the vehicle: The driver's longitudinal driving intention is inferred through the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus. The driver's longitudinal driving intention includes but is not limited to following the vehicle, accelerating, braking, stopping and holding;

[0047] Step 3: Understanding other drivers’ driving intentions: Infer the driver’s other driving intentions through the driver’s head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus. The driver’s other driving intentions include but are not limited to hesitation and overtaking.

[0048] The above steps 1, 2 and 3 are for understanding different aspects of the driver's driving intention, which are decoupled from each other and do not interfere with each other. They are judged in parallel during reasoning. The judgments of steps 1, 2 and 3 are similar. The methods of steps 2 and 3 are similar to those of step 1.

[0049] Step 5: Generate human-machine hybrid enhanced perception map. The specific steps are as follows:

[0050] Step 1: Modeling the driver’s perception target: In most cases, the driver can correctly control driving. Therefore, the first step for the machine and driver to achieve fusion perception should be to understand the driver’s perception target. The driver’s perception target modeling includes the following steps:

[0051] Step 1: Modeling the driver's gaze direction: Model the driver's current gaze direction vector based on the detected head movement signal and eye movement signal. The gaze direction vector includes the position of the driver's eye center in the vehicle coordinate system (xe, ye, ze) and the unit vector of the gaze direction in the vehicle coordinate system (xd, yd, zd);

[0052] Step 2: Driver’s Region of Interest (ROI) Judgment: When the driver’s gaze direction remains in the same or similar area for a period of time, the area can be considered as the driver’s region of interest, and the objects in the driver’s region of interest are the driver’s perception targets;

[0053] Step 3: Modeling the driver's perception target: After the driver's perception target is confirmed, it needs to be displayed in the human-machine hybrid enhanced perception map in step 4. Therefore, it is necessary to establish a structure sequence to store the output target information of the perception map. In the human-machine hybrid enhanced perception map, the object's ID, relative position, relative speed and TTC are recorded and output in the human-machine hybrid enhanced perception map in step 4;

[0054] Relative position refers to the shortest distance of an object relative to the edge of the vehicle, rather than the position of the object in the vehicle coordinate system, in order to adapt to the driver's perception habits. Relative speed is expressed as the sum of speeds, and the positive and negative speed values ​​are defined as follows: when the angle between the relative speed vector and the main vehicle speed vector is acute, it is positive, and when the angle is obtuse, it is negative;

[0055] The driver's attention targets are analyzed based on the previously calculated driving situation, and flags are set to mark targets that seriously affect the current driving situation. A flag of 0 indicates a moderate impact; a flag of 1 indicates a large impact and requires special attention; a flag of 2 indicates a very large impact and an emergency, which also indicates that the current driving situation is judged to be dangerous;

[0056] Step 2: Modeling the driver’s perception blind spots: The driver’s perception is limited. There are many blind spots that the driver cannot perceive during driving. The information of these blind spots will often become the key to avoiding accidents. The driver’s perception blind spots mainly include:

[0057] Front blind spot: refers to the area in front of the hood, caused by the height of the vehicle body and the length of the front of the vehicle;

[0058] Rear blind spot: refers to the area behind the vehicle that cannot be seen by the rearview mirror and reflector. When modeling, it is approximately considered that all areas behind the vehicle are treated as blind spots because they cannot be adequately observed;

[0059] Rearview mirror blind spot: refers to the area on both sides of the vehicle that is included due to the limited field of view of the rearview mirror. The AB pillar blind spot is the area on both sides of the windshield that cannot be observed due to the obstruction of the AB pillars.

[0060] If there is an object within the above-mentioned blind spot area, the driver will be informed as key information. When modeling, a spatial position model of the area covered by the blind spot is established based on the principle that light propagates in a straight line;

[0061] Step 3: Machine perception map modeling: The machine perception map mainly includes the parts that the driver cannot actively perceive and require machine assistance, mainly including blind spot perception map modeling, driver inattention perception map modeling and demand perception map based on driver driving intention; blind spot perception map modeling refers to screening out all objects in the blind spot by comparing the spatial position of objects in the surrounding driving environment with the spatial position of the blind spot, recording the object's ID, relative position, relative speed and TTC information, and calculating the severity of the impact of the object on the current driving situation, and then sending it to the human-machine hybrid enhanced perception map in step 4. The driver inattention perception map refers to objects in the surrounding driving environment that are not in the blind spot or in the driver's area of ​​interest, but should be noticed by the driver. The screening of such objects is achieved by calculating the object's The impact severity mark of the current driving situation is completed. After calculation, the ID, relative position, relative speed, TTC and severity mark of the objects with marks greater than 0 are recorded and sent to the human-machine hybrid enhanced perception map in step 4, similar to the previous one. The demand perception map based on the driver's driving intention means that when the driver completes a driving operation, he often needs to perceive multiple targets. When the driver changes lanes, he needs to observe the vehicle in front of him and the vehicle behind him. In addition to the area of ​​interest described in step 1, he also needs to perceive objects at specific locations according to the driver's driving intention. The input of this link is the three types of driver driving intentions obtained in step 4. The perception results of the above-mentioned objects at specific locations are obtained by using a high-dimensional space search algorithm, and are presented to the driver through the HMI as output.

[0062] Step 4: Modeling of human-machine hybrid enhanced perception map: The human-machine hybrid enhanced perception map is the output of the entire system and is provided to both the driver and the intelligent driving system for perception result output. The perception map provided to the driver mainly includes the quantitative data information of all objects perceived in steps 1, 2 and 3 in the fifth step that need to be displayed to the driver, the traffic light information ahead and the traffic sign information of the current driving road. The perception map provided to the intelligent driving system includes all the information of all objects in the surrounding driving environment, driving situation information, and the driver's driving intention information. The output links of the human-machine hybrid enhanced perception map to the driver are as follows:

[0063] Link 1: Output of perception map to the driver: The ID, relative position, relative speed and TTC information of the driver's perceived target are projected directly on the front windshield through HUD head-up display technology, and displayed in the direction of the driver's perceived target; the same treatment is performed for the objects in the driver's negligent perception map identified in step 3 of step 5; if the object in the blind spot perception map identified in step 3 of step 5 exists, it will be prompted in the form of a warning at the bottom of the front windshield. In addition to the conventional ID, relative speed and TTC information, the warning content also needs to include the blind spot position to replace the position relative to the main vehicle. This is more intuitive and can more effectively reduce the driver's perception burden and achieve a good human-machine enhanced perception effect;

[0064] For the above-mentioned atlas display, different colors are displayed according to the position of the severity mark of the impact of the object on the current driving situation in the perception atlas. For objects with a severity mark of 0, green is displayed, for objects with a severity mark of 1, orange is displayed, and for objects with a severity mark of 2, red is displayed;

[0065] When displaying, if there are two or more objects that overlap, the severity flag of the impact of the object on the current driving situation is used to judge, and the object with a larger severity flag value is displayed first; if there are two or more objects that overlap and have the same severity flag value, they are sorted according to the relative distance, and the object with a closer relative distance is displayed first;

[0066] In addition, traffic light signals and traffic sign signals are displayed on the top of the front windshield. When the driver violates traffic regulations, the top display will flash brightly to warn the driver.

[0067] Link 2: Perception map output to human-machine co-driving intelligent driving system: As the perception layer of human-machine co-driving technology, the human-machine hybrid enhanced perception map needs to output relevant information of the driver, including the driver's driving intention. All information about objects in the surrounding driving environment and driving situation information serve as calculation input and output for subsequent planning, decision-making and control.

[0068] Beneficial effects of the present invention:

[0069] The human-machine fusion perception method described in the present invention has no requirements for driver status, driver type, vehicle type, etc., and is universal; all modules of the present invention have achieved driver-in-the-loop, realizing the organic combination of human drivers and intelligent driving systems; the human-machine fusion perception method described in the present invention can fully recognize the driver, thereby providing personalized fusion perception map output according to the driver's status, perception target and other information, reducing information complexity, reducing the driver's perception burden, and not missing key information, thereby ensuring the safety of the overall system; the human-machine fusion perception method described in the present invention is also compatible with different human-machine co-driving decision-making planning and control systems, and the rich information interface can meet the needs of most human-machine co-driving decision-making planning and control systems; the present invention can combine advanced technical means from multiple disciplines, conform to the development trend of autonomous driving technology, and has broad application prospects and feasibility in multiple scenarios and multiple working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of the human-machine fusion perception method described in the present invention.

[0071] Figure 2 This is a schematic diagram of the framework of the human-machine fusion perception method described in the present invention.

[0072] Figure 3 This is a schematic diagram of an exemplary architecture of the first step of the present invention.

[0073] Figure 4 This is a schematic diagram of an exemplary architecture of the second step described in the present invention.

[0074] Figure 5 This is a schematic diagram of an exemplary architecture of the third step described in the present invention.

[0075] Figure 6 This is a schematic diagram of an exemplary architecture of the fourth step described in the present invention.

[0076] Figure 7 This is a schematic diagram of an exemplary architecture of the fifth step described in the present invention.

[0077] Figure 8 It is a schematic diagram of the structure of the convolutional neural network described in the present invention.

[0078] Fig. 9 Schematic diagram of the algorithm flow of the self-attention mechanism described in the present invention.

[0079] Fig.10 This is a logical diagram of the driving intention understanding according to the present invention.

[0080] Fig.11 Schematic diagram of the blind area described in the present invention.

[0081] Fig.12 This is a schematic diagram of the results of the human-machine hybrid enhanced atlas HUD display described in the present invention.

[0082] Fig.13 This is a schematic diagram of information transmission between the human-machine hybrid enhanced perception map and the human-machine co-driving intelligent driving system described in the present invention. DETAILED DESCRIPTION

[0083] See also Figures 1 to 13 As shown:

[0084] The human-machine fusion perception method for highly automated driving provided by the present invention comprises the following steps:

[0085] Step 1: Data preprocessing

[0086] Step 2: Driving situation awareness

[0087] Step 3: Extraction of typical biological signals of drivers

[0088] Step 4: Understanding the driver’s driving intention

[0089] Step 5: Human-machine hybrid enhanced perception map generation

[0090] The process of data preprocessing in the first step is as follows:

[0091] Step 1: Collect data. Collect the on-board CAN bus, camera sensor signals, laser radar millimeter-wave radar signals, and vehicle-road cooperative equipment signals carried by the intelligent vehicle, and collect the following sensor data according to a fixed sampling period: vehicle control and motion state data represented by time series generated on the on-board CAN bus, including clutch, brake, and accelerator pedal control signals, transmission control signals, and steering wheel control signals; color image streams with a certain frame rate generated by each camera sensor. The camera sensors mainly include camera sensors distributed in front of the driver to identify the driver's status and camera sensors distributed around the vehicle to identify the surrounding traffic environment. They output color image streams with driver status information and color image streams with surrounding traffic environment information respectively; a list of surrounding object motion states generated by each laser radar millimeter-wave radar sensor at a certain frame rate. Millimeter-wave radars are mainly distributed around the vehicle and output a list of surrounding object motion states with surrounding traffic environment information; a list of real-time traffic status signals transmitted by vehicle-road cooperative equipment.

[0092] Step 2: Verify and align the data. This step includes:

[0093] Step 1: Verify the validity of sensor data. Read the values ​​of the vehicle bus signal, vehicle state measurement signal, and multimodal environment sensor signal respectively and evaluate whether they are collected and generated when the sensor is working normally. Use an integer mark to represent it. A positive value indicates that the sensor is working normally, and a negative value indicates an abnormality. If the input signal is judged to be abnormal, set the integer mark to a negative number in the output signals of the first step, namely, the vehicle bus alignment signal, the vehicle state measurement alignment signal, and the multimodal environment sensor alignment signal.

[0094] Step 2: Time alignment. Time alignment refers to unifying the data collected by different sensors in different time coordinate systems into the same time coordinate system based on the physical hardware information of the sensor, such as the frame rate and startup time of the camera sensor and the millimeter wave radar sensor. This allows the data from different sensors to be interconnected, which is beneficial for subsequent data analysis.

[0095] Time alignment strategies include, but are not limited to: setting up a common data buffer to store different sensor data; calculating absolute time based on information such as frame rate and startup time; or setting system relative time within the intelligent vehicle system to set a global timestamp for all sensor data.

[0096] Step 3: Spatial alignment. Similar to time alignment, sensor spatial alignment means unifying the data collected by different sensors in their respective coordinate systems into the same coordinate system, the vehicle coordinate system, based on the sensor placement information. This allows the data from different sensors to be interconnected, which is beneficial for subsequent data analysis.

[0097] Step 4: Integrate the aligned data. The sensor data after time alignment and space alignment is actually a traffic environment image stream with the motion status of surrounding objects and a driver status color image stream in a unified time and space coordinate system. These two parts are integrated into a structure of unified format and sent to the second and third steps as their data input.

[0098] The process of extracting typical biological signals of drivers in the second step is as follows:

[0099] Step 1: Head movement signal recognition: Head movement signal recognition mainly recognizes the three posture angles (Yaw, Roll, Pitch) of the driver's head posture and the spatial position of the eyes.

[0100] Step 2: Eye movement signal recognition. Eye movement signal recognition mainly recognizes eye movement vector biological signals. By identifying the driver's eye movement vector, the driver's gaze behavior can be modeled to obtain the driver's gaze direction.

[0101] The head movement recognition method described in step 1 is a convolutional neural network. As a mainstream deep learning method, convolutional neural networks are widely used in the field of computer vision image recognition and have the advantages of fast speed and high accuracy.

[0102] The convolutional neural network used in this step is as follows Figure 8 As shown, the specific structure is as follows:

[0103] The input image is cropped to a 256*256*3 image, and then processed by the first convolution layer (convolution kernel size is 3*3*3, convolution step size is 2), the first pooling layer (using the maximum pooling method, size 3*3, step size 2), the second convolution layer (convolution kernel size is 3*3*32, step size is 2), and the second pooling layer (using the maximum pooling method, size 2*2, step size 2) to obtain a 16*16*128 feature layer. After passing through a neural layer composed of 128 neurons and normalized by the softmax algorithm, the coordinate positions of each feature point on the face can be obtained. The feature points mentioned above include eyebrows, corners of the eyes, cornea, pupils, nose tip, nostrils, mouth, chin, and ears.

[0104] According to the positions of the above-mentioned feature points, the driver's head posture can be estimated by using the active appearance model (AAM) method, thereby obtaining the driver's head movement signal, which will serve as part of the input for understanding the driver's driving intention in the fourth step.

[0105] The eye movement signal recognition described in step 2 is similar to step 1. It mainly uses the cornea and pupil feature points output by the convolutional neural network. The position of the same feature point at the same time collected by the binocular camera is used for spatial modeling to obtain a three-dimensional eyeball model, thereby calculating the position of the eye in space and the eye movement vector. The calculated eye movement signal is passed as input to the fourth step and the driver's driving intention understanding and the fifth step of human-machine hybrid enhanced perception map generation.

[0106] The process of driving situation awareness in the third step is as follows:

[0107] Step 1: Driving environment perception. Using the traffic environment information around the vehicle transmitted by each sensor after preprocessing in the first step, the necessary traffic participants are extracted to form a typical traffic condition. The established traffic environment should include relevant information of all traffic participants that may interact with the main vehicle in a short period of time, including category (people, cars, cones, etc.), relative distance (including lateral distance and longitudinal distance), and relative speed.

[0108] In addition, the first step of driving environment perception also requires the recognition of road information such as lane lines and traffic signs. The recognition methods include but are not limited to: fitting lane lines, identifying traffic signs and traffic light signals using computer vision-based image recognition algorithms; transmitting lane, traffic sign and traffic light information through vehicle-road collaboration-based intelligent transportation system vehicle-road equipment communication; or reading traffic road information based on high-precision map information in the cloud.

[0109] Step 2: Understanding driving scenarios. The driving scenario is defined as the external environment in which the driver obtains driving-related information at the current moment. Understanding the driving scenario mainly refers to the extraction of the current driving conditions based on the driving environment built in step 1. Typical driving scenarios include but are not limited to: smooth urban roads, congested urban roads, intersections, site parks, highways, viaducts, etc. The extraction method can use feature comparison to divide different driving scenarios according to the values ​​of key parameters such as the density of traffic participants on the road, speed, shape of the road, and the location of traffic lights.

[0110] Step 3: Driving situation reasoning. Driving situation refers to the driving environment status and development trend that can affect the driver and the autonomous driving system over a period of time. The purpose of driving situation reasoning is to comprehensively consider the current driving environment status and give the most appropriate perception map after comprehensively considering the driver's status, so as to achieve the effect of fusion enhanced perception.

[0111] In step 1, the following scheme can be used to extract traffic participants:

[0112] Solution 1: Safety time algorithm. This algorithm is a forward collision warning algorithm based on the collision time between the following vehicle and the preceding vehicle. The collision time calculation formula is:

[0113] TTC=D / (V r -V f )

[0114] Where D is the distance between the following vehicle and the preceding vehicle; V r Indicates the speed of the following vehicle; V f Indicates the speed of the preceding vehicle. This algorithm can be used to extract traffic participants on the same lane line.

[0115] For traffic participants on different lanes, the collision time calculation formula changes to:

[0116] TTC=D / v

[0117] Where D is the distance between the expected collision position of the two vehicles and the position of the vehicle; v is the speed of the vehicle.

[0118] By setting the TTC threshold, the traffic participants that need to be considered in the surrounding environment can be extracted. This patent recommends taking 5s under low-speed conditions (below 40km / h), 7.5s under medium-speed conditions (below 80km / h), and 10s under high-speed conditions (above 80km / h).

[0119] Solution 2: Safety distance algorithm. In the actual traffic environment, there are some traffic participants that will not pose a safety risk to the vehicle, but can affect the driving of the main vehicle. There is a certain relationship between the absolute distance between these traffic objects and the main vehicle. Define the safety distance DS:

[0120] DS = vm * 2

[0121] Where, vm = max{v e , v o , 10}, unit is km / h; v e is the speed of the main vehicle, in km / h; v o is the speed of surrounding objects, in km / h; DS is in m.

[0122] When DS is smaller than the actual distance between the host vehicle and the object, it indicates that the object is within the safe distance range, will affect the driving of the host vehicle, and should be included in the traffic environment modeling.

[0123] The input of the driving situation reasoning model in step 3 is the driving situation history data generated in step 2. The driving situation reasoning method is the Sentiment Analysis method of deep learning. The algorithm design structure is as follows: Fig. 9 In order to quantitatively consider the driving situation, the driving situation is quantitatively represented by SA = [sa1, sa2], where sa i The value range is 1-5, sa1 indicates a traffic crisis situation, sa1=1 indicates that there is no safety risk in the environment, and gradually increases to sa1=5, indicating that the vehicle is about to have an inevitable traffic accident; sa2 indicates the current level of traffic congestion, sa2=1 indicates that the road is unobstructed, and gradually increases to sa2=5, indicating that the vehicle is in severe congestion.

[0124] The process of driver's driving intention understanding in the fourth step is as follows:

[0125] Step 1: Understanding the vehicle's lateral driving intention. The driver's lateral driving intention is inferred through the driver's head movement signals, eye movement signals, driving environment, and driving operation signals transmitted by the vehicle's CAN bus. The driver's lateral driving intention includes but is not limited to turning, changing lanes, and cutting in.

[0126] Step 2: Understanding the longitudinal driving intention of the vehicle. The driver's longitudinal driving intention is inferred through the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus. The driver's longitudinal driving intention includes but is not limited to following the vehicle, accelerating, braking, stopping, and holding.

[0127] Step 3: Understanding other drivers’ driving intentions. The driver’s other driving intentions are inferred through the driver’s head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus. The driver’s other driving intentions include but are not limited to hesitation and overtaking.

[0128] Step 1, step 2, and step 3 are for understanding different aspects of the driver's driving intentions. They are decoupled from each other and do not interfere with each other, and are judged in parallel during reasoning.

[0129] Since the judgments of step one, step two, and step three are similar, only step one is described in detail, and the methods of step two and step three are similar to step one.

[0130] The understanding of the vehicle's lateral driving intention described in step 1 is established using a hidden Markov model. The hidden Markov model solves the problem of understanding the driver's lateral driving intention in three steps:

[0131] Step 1: Model data definition and collection. The problem of understanding the driver's lateral driving intention corresponds to the evaluation problem of the hidden Markov model, that is, the problem of estimating the probability value. The core of this problem is to find the probability of the driver's lateral driving intention being a certain value under a certain traffic situation and driver state.

[0132] We denote the driver’s lateral driving intention as Q, which is defined as follows:

[0133] Q=-3, -2, -1, 0, 1, 2 respectively indicate that the driver's current lateral driving intention is to turn around, turn left, change lane to the left, keep going straight, change lane to the right, and turn right.

[0134] We take the driver's head movement signal, eye movement signal, driving environment and driving operation signal transmitted by the vehicle CAN bus as observation variable input, and obtain the observation variable sequence O1, O2, . . . , O3 after time discretization. t We take the driver's lateral driving intention as the hidden state variable of the hidden Markov model, and obtain the hidden state variable sequence Q1, Q2, . . . , Q after time discretization. t .

[0135] The parameters of the hidden Markov model are defined as λ = (π, A, B), where π represents the initial state probability of the system, A represents the state transfer matrix, and B represents the emission matrix. Initial state probability π: The probability of each state of the model appearing at the initial moment, usually recorded as π = (π1, π2, ..., π t ), π i Indicates the probability that the initial observed variable data of the model is Oi. State transition probability A: The probability of the model switching between various hidden states, usually recorded as matrix A = [a ij ], where a ij It means that at any time t, if the hidden state variable data is Q i , then at the next moment the hidden state variable data is Q j The probability of the emission matrix B is the probability relationship between the model observation variables and the hidden state variables, usually recorded as matrix B = [b jk ], where b jk It means that at any time t, if the observed variable data is O j , then at the current moment the hidden state variable data is Q k probability.

[0136] Furthermore, according to the parameters defined above, experiments are conducted to collect a sufficiently large data set, and then the data set is initialized and grouped according to the K-means clustering algorithm to obtain the initial parameter λ0 of the model.

[0137] Step 2: Model training. Model training refers to repeated deep learning training based on the obtained data set and the principle of hidden Markov model to improve the accuracy of the model so that it can normally understand the driver's lateral driving intention.

[0138] The main content of model training is: using the forward-backward algorithm to calculate the observed variable data used in the current training in the collected data set train The output probability P(O train |λ). Apply the EM algorithm to update the model parameter λ, specifically including calculating the new state transfer matrix A and the new emission matrix B. Repeat the training by replacing the observed variable data and the corresponding hidden state variable data, and train |λ) is monitored, and when the value converges to a very small change or no longer changes, the training ends, and the hidden Markov model parameter λ obtained after the last training is end As the final hidden Markov model parameters, a hidden Markov model that can be used to understand the actual driver's lateral driving intention is obtained.

[0139] Step 3: Understanding the driver's lateral driving intention. After training, the hidden Markov model can be used to understand the driver's lateral driving intention. The understanding process is the decoding problem solving process of the hidden Markov model, such as Fig.10 The decoding process uses the Viterbi algorithm to discretize the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus over a period of time to obtain the observed variable sequence O1,...,O t Input the model and find the probability that the driver's lateral driving intention Q at the next moment is -3, -2, -1, 0, 1, and 2 respectively. The one with the largest probability is taken as the final output of the driver's lateral driving intention.

[0140] The process of generating the human-machine hybrid enhanced perception map in the fifth step is as follows:

[0141] Step 1: Modeling the driver’s perception target. In most cases, the driver can correctly control driving. Therefore, the first step in the fusion perception of the machine and the driver should be to understand the driver’s perception target.

[0142] Step 2: Modeling the driver's perception blind spots. The driver's perception is limited, and there are many blind spots that the driver cannot perceive during vehicle driving. The information of these blind spots will often become the key to avoiding accidents.

[0143] Step 3: Machine perception map modeling. The machine perception map mainly includes the parts that the driver cannot actively perceive and need machine assistance to perceive. It mainly includes blind spot perception map modeling, driver negligence perception map modeling, and demand perception map based on driver driving intention.

[0144] Step 4: Modeling of human-machine hybrid enhanced perception map. The human-machine hybrid enhanced perception map is the output of the entire system and is provided to both the driver and the intelligent driving system as the perception result output. The perception map provided to the driver mainly includes the quantitative data information of all objects perceived in steps 1, 2, and 3 that need to be displayed to the driver, the traffic light information ahead, and the traffic sign information of the current road. The perception map provided to the intelligent driving system includes all the information of all objects in the surrounding driving environment, driving situation information, and the driver's driving intention information.

[0145] The driver perception target modeling described in step 1 includes the following steps:

[0146] Step 1: Modeling the driver's gaze direction. Model the driver's current gaze direction vector based on the driver's head movement signal and eye movement signal detected in step 1. The gaze direction vector includes the position of the driver's eye center in the vehicle coordinate system (xe, ye, ze) and the unit vector of the gaze direction in the vehicle coordinate system (xd, yd, zd).

[0147] Step 2: Determine the driver's region of interest (ROI). When the driver's gaze direction remains in the same or similar area for a period of time, the area can be considered as the driver's region of interest, and the objects in the driver's region of interest are the driver's perception targets.

[0148] Step 3: Modeling the driver's perception target. After the driver's perception target is confirmed, it needs to be displayed in the human-machine hybrid enhanced perception map in step 4. Therefore, a structure sequence needs to be established to store the output target information of the perception map. Compared with human drivers, one of the advantages of machine perception is that it can obtain quantitative results accurate to numbers. In the human-machine hybrid enhanced perception map, we record the object's ID, relative position, relative speed, and TTC, and output it in the human-machine hybrid enhanced perception map in step 4.

[0149] In particular, the relative position refers to the shortest distance of the object relative to the edge of the vehicle, rather than the position of the object in the vehicle coordinate system, in order to adapt to the driver's perception habits. The relative speed is expressed as the sum of the speeds, and the positive and negative speed values ​​are defined as follows: when the angle between the relative speed vector and the host vehicle speed vector is acute, it is positive, and when the angle is obtuse, it is negative.

[0150] Furthermore, we can analyze the driver's focus targets based on the previously calculated driving scenarios, and set flags to mark targets that seriously affect the current driving scenario. A flag of 0 indicates a general impact; a flag of 1 indicates a large impact and requires special attention; a flag of 2 indicates a very large impact and an emergency, which also indicates that the driving scenario is judged to be dangerous at the current moment.

[0151] The driver's perception blind spots described in step 2 mainly include:

[0152] The front blind spot refers to the area in front of the hood, which is caused by factors such as the height of the vehicle body and the length of the front of the vehicle. The rear blind spot refers to the area behind the vehicle that cannot be seen by the rearview mirror and the reflector. When modeling, it can be roughly considered that all areas behind the vehicle are treated as blind spots because they cannot be observed enough. The rearview mirror blind spot refers to the area on both sides of the vehicle that is included due to the limited field of view of the rearview mirror. The AB pillar blind spot is the area on both sides of the windshield that cannot be observed due to the AB pillars.

[0153] The specific range of the blind area is as follows: Fig.11 shown.

[0154] If there are objects within the above-mentioned blind spot area, they should be informed to the driver as key information. When modeling, a spatial position model of the area covered by the blind spot can be established based on the principle that light propagates in a straight line.

[0155] The blind spot perception map modeling described in step 3 refers to screening out all objects in the blind spot by comparing the spatial position of objects in the surrounding driving environment with the spatial position of the blind spot, recording the object's ID, relative position, relative speed, TTC information, and calculating the severity of the impact of the object on the current driving situation. Then it is sent to the human-machine hybrid enhanced perception map in step 4.

[0156] The driver's inattention perception map described in step 3 refers to objects in the surrounding driving environment that are not in the blind spot or the driver's area of ​​interest, but should be noticed by the driver. The screening of such objects can be completed by calculating the severity of the impact of the object on the current driving situation. After the calculation, the ID, relative position, relative speed, TTC and severity flag of the object with a flag greater than 0 are recorded. Similar to the previous one, the human-machine hybrid enhanced perception map is sent to step 4.

[0157] Step 3 describes the demand-based perception map based on the driver's driving intention. When a driver completes a driving maneuver, he often needs to perceive multiple targets. For example, when changing lanes, the driver needs to observe the vehicle in front of him and the vehicle behind him. In other words, in addition to the area of ​​interest described in step 1, objects at specific locations need to be perceived based on the driver's driving intention. The input of this link is the three types of driver driving intentions obtained in step 4. The perception results of the above-mentioned objects at specific locations can be obtained by using a high-dimensional space search algorithm, and presented to the driver as output through the HMI.

[0158] The output of the human-machine hybrid enhanced perception map to the driver described in step 4 is as follows:

[0159] Step 1: Output of the perception map to the driver. The HUD head-up display technology is directly projected onto the front windshield to display the ID, relative position, relative speed, and TTC information of the driver's perceived target in the direction of the driver's perceived target; the same treatment is applied to the objects in the driver's negligent perception map identified in step 3; if the object in the blind spot perception map identified in step 3 exists, a warning will be displayed at the bottom of the front windshield. In addition to the conventional ID, relative speed, and TTC information, the warning content also needs to include the blind spot position to replace the position relative to the main vehicle. This is more intuitive and can more effectively reduce the driver's perception burden and achieve a good human-machine enhanced perception effect.

[0160] Furthermore, for the above-mentioned map display, different colors are displayed according to the severity flag value of the impact of the object in the perception map on the current driving situation. For objects with a severity flag of 0, green is displayed, for objects with a severity flag of 1, orange is displayed, and for objects with a severity flag of 2, red is displayed.

[0161] When displaying, if there are two or more objects overlapping, the severity flag of the impact of the object on the current driving situation is used for judgment, and the object with a larger severity flag value is displayed first; if there are two or more objects overlapping and the severity flag values ​​are the same, they are sorted according to the relative distance, and the object with a closer relative distance is displayed first.

[0162] In addition, traffic light signals and traffic sign signals are displayed on the top of the front windshield. When the driver violates traffic regulations, the top display will flash brightly to warn the driver.

[0163] An example of a human-machine hybrid enhanced perception map HUD display is shown below: Fig.12 shown.

[0164] Link 2: The output of the perception map to the human-machine co-driving intelligent driving system. The output of the human-machine hybrid enhanced perception map to the intelligent driving system is based on the requirements of the intelligent driving system. With the development of human-machine co-driving technology, the planning, decision-making and control layers of the human-machine co-driving intelligent driving system are increasingly considering the influence of driver personalization factors. For this reason, the human-machine hybrid enhanced perception map, as the perception layer of human-machine co-driving technology, needs to output relevant information of the driver, such as the driver's driving intention. Other driver status quantities such as driver fatigue status and driver personality are not further discussed in this patent and can be filled in according to the needs of the human-machine co-driving intelligent car framework. The recognition method is similar to the driver's driving intention. All information about objects in the surrounding driving environment and driving situation information are used as calculation input and output for subsequent planning, decision-making and control.

[0165] Attached Fig.13 Information transmission for human-machine hybrid enhanced perception map and human-machine co-driving intelligent driving system.

Claims

1. A human-machine fusion perception method for highly automated driving, characterized by: The method comprises the following steps: The first step is data preprocessing. The specific steps are as follows: Step 1, data collection: collect the on-board CAN bus, camera sensor signals, laser radar millimeter-wave radar signals, and vehicle-road cooperative equipment signals carried by the intelligent vehicle, and collect the following sensor data according to a fixed sampling period: vehicle control and motion state data represented by time series generated on the on-board CAN bus, including clutch, brake, and accelerator pedal control signals, transmission control signals, and steering wheel control signals; color image streams with a certain frame rate generated by each camera sensor. The camera sensors mainly include camera sensors distributed in front of the driver to identify the driver's status and camera sensors distributed around the vehicle to identify the surrounding traffic environment. They output color image streams with driver status information and color image streams with surrounding traffic environment information respectively; a list of surrounding object motion states generated by each laser radar millimeter-wave radar sensor at a certain frame rate. Millimeter-wave radars are mainly distributed around the vehicle and output a list of surrounding object motion states with surrounding traffic environment information; a list of real-time traffic status signals transmitted by the vehicle-road cooperative equipment; Step 2: Verify and align the data: The specific steps are as follows: Step 1: Verify the validity of sensor data: Read the values ​​of the vehicle bus signal, vehicle status measurement signal, and multimodal environmental sensor signal respectively and evaluate whether they are collected and generated when the sensor is working normally. Use an integer mark to represent them. A positive value indicates that the sensor is working normally, and a negative value indicates an abnormality. If it is determined that the input signal is abnormal, then the integer flag is set to a negative number in the output signals of the first step, namely, the vehicle bus alignment signal, the vehicle state measurement alignment signal, and the multimodal environment sensor alignment signal; Step 2: Time alignment: Time alignment refers to unifying the data collected by different sensors in different time coordinate systems into the same time coordinate system based on the sensor physical hardware information, which refers to the frame rate and startup time of the camera sensor and the millimeter wave radar sensor. This makes the data of different sensors interconnected, which is conducive to subsequent data analysis. The strategies for time alignment include but are not limited to: setting a common data buffer to store different sensor data; calculating the absolute time based on the frame rate and start time information; or setting the system relative time inside the intelligent vehicle system to set a global timestamp for all sensor data; Step 3: Spatial alignment: Sensor spatial alignment means unifying the data collected by different sensors in their respective coordinate systems into the same coordinate system, i.e. the vehicle coordinate system, based on the sensor placement information. This allows the data from different sensors to be interconnected, which is beneficial for subsequent data analysis. Step 4: Integrate the aligned data: After time alignment and space alignment, the sensor data is actually a traffic environment image stream with the motion status of surrounding objects and a driver status color image stream in a unified time-space coordinate system. These two parts are integrated into a structure of a unified format and sent to the second and third steps as their data input; The second step is driving situation awareness. The specific steps are as follows: Step 1: Head movement signal recognition: Head movement signal recognition mainly identifies the three posture angles of the driver's head posture and the spatial position of the eyes. The head movement recognition method uses a convolutional neural network. The specific structure of the convolutional neural network is as follows: The input image is cropped into an image of 256*256*3, and processed in sequence by the first convolution layer, the first pooling layer, the second convolution layer, and the second pooling layer to obtain a 16*16*128 feature layer. Then, it passes through a neural layer composed of 128 neurons and is normalized by the softmax algorithm to obtain the coordinate positions of the facial feature points. The convolution kernel size of the first convolution layer is 3*3*3, and the convolution step size of the first convolution layer is 2; the first pooling layer adopts the maximum pooling method, and the size of the first pooling layer is 3*3, and the first pooling step size is 2. The stride of the layer is 2; the convolution kernel size of the second convolution layer is 3*3*32, and the stride of the second convolution layer is 2; the second pooling layer adopts the maximum pooling method, the size of the second pooling layer is 2*2, and the stride of the second pooling layer is 2; the aforementioned feature points include eyebrows, eye corners, corneas, pupils, nose tips, nostrils, mouths, chins and ears. According to the positions of the feature points, the active appearance model method can be used to estimate the driver's head posture, thereby obtaining the driver's head movement signal, which will be used as part of the input for understanding the driver's driving intention in the fourth step; Step 2, eye movement signal recognition: Eye movement signal recognition mainly recognizes eye movement vector biological signals. By identifying the driver's eye movement vector, the driver's gaze behavior is modeled to obtain the driver's gaze direction. Eye movement signal recognition is similar to step 1. It mainly uses the cornea and pupil feature points output by the convolutional neural network. The position of the same feature point at the same time collected by the binocular camera is used for spatial modeling to obtain a three-dimensional eyeball model, thereby calculating the position of the eye in space and the eye movement vector. The calculated eye movement signal is passed as input to the fourth step and the driver's driving intention understanding and the fifth step of human-machine hybrid enhanced perception map generation; The third step is to extract typical biological signals of drivers. The specific steps are as follows: Step 1: Driving environment perception: Use the traffic environment information around the vehicle transmitted by the sensors after the preprocessing in the first step to extract the necessary traffic participants to form a typical traffic condition. The established traffic environment includes the relevant information of all traffic participants that may interact with the main vehicle in a short period of time, including the category, relative distance and relative speed; Driving environment perception also requires the recognition of lane lines and traffic sign road information. The recognition methods include but are not limited to: fitting lane lines based on computer vision image recognition algorithms, identifying traffic signs and traffic light signals; intelligent transportation system based on vehicle-road collaboration, vehicle-roadbed equipment communication to transmit lane, traffic sign and traffic light information; or traffic road information reading based on cloud-based high-precision map information; In order to realize the extraction of traffic participants, the following schemes are used for calculation: Solution 1: Safety time algorithm: This algorithm is a forward collision warning algorithm based on the collision time between the following vehicle and the preceding vehicle. The collision time calculation formula is as follows: TTC=D / (V r -V f ) Where D is the distance between the following vehicle and the preceding vehicle; V r Indicates the speed of the following vehicle; V f Indicates the speed of the preceding vehicle. This algorithm is used to extract traffic participants on the same lane line; For traffic participants on different lanes, the collision time calculation formula changes to: TTC=D / v Where D is the distance between the expected collision position of the two vehicles and the position of the vehicle; v is the speed of the vehicle; Setting the TTC threshold can extract the traffic participants that need to be considered in the surrounding environment; Solution 2: Safety distance algorithm: In the actual traffic environment, there are some traffic participants that will not pose a safety risk to the vehicle, but can affect the driving of the main vehicle. There is a certain relationship between the absolute distance between these traffic objects and the main vehicle. The safety distance DS is defined. The specific formula is as follows: DS = vm * 2 Where, vm = max{v e , v o , 10}, unit is km / h; v e is the speed of the main vehicle, in km / h; v o is the speed of surrounding objects, in km / h; DS is in m; When DS is less than the actual distance between the host vehicle and the object, it indicates that the object is within the safe distance range and will affect the driving of the host vehicle, and should be included in the traffic environment modeling; Step 2: Understanding driving scenarios: Driving scenarios are defined as the external environment in which the driver obtains driving-related information at the current moment. Understanding driving scenarios mainly refers to the extraction of current driving conditions based on the driving environment built in step 1. Typical driving scenarios include but are not limited to: unobstructed urban roads, congested urban roads, intersections, site parks, highways and viaducts. The extraction method uses feature comparison to divide different driving scenarios according to the values ​​of key parameters such as the density of traffic participants on the road, speed, road shape and location of traffic lights. Step 3: Driving situation reasoning: Driving situation refers to the driving environment status and development trend that can affect the driver and the automatic driving system over a period of time. The purpose of driving situation reasoning is to comprehensively consider the current driving environment status and give the most appropriate perception map after comprehensively considering the driver's status, so as to achieve the effect of fusion enhanced perception. The input of the driving situation reasoning model is the driving situation historical data generated in step 2. The driving situation reasoning method is the Sentiment Analysis method of deep learning, which quantitatively represents the driving situation. The representation method is SA = [sa1, sa2], where sa i The value range is 1-5, sa1 indicates the traffic crisis situation, sa1=1 means there is no safety risk in the environment, and gradually increases to sa1=5, indicating that the vehicle is about to have an unavoidable traffic accident; sa2 indicates the current traffic congestion level, sa2=1 means the road is unobstructed, and gradually increases to sa2=5, indicating that the vehicle is in severe congestion; Step 4: Understanding the driver's driving intention. The specific steps are as follows: Step 1: Understanding the vehicle's lateral driving intention: The driver's lateral driving intention is inferred through the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle's CAN bus. The driver's lateral driving intention includes but is not limited to turning, changing lanes and cutting in; the vehicle's lateral driving intention understanding is established using a hidden Markov model. The hidden Markov model solves the problem of understanding the driver's lateral driving intention in the following three steps: Step 1: Model data definition and collection: The problem of understanding the driver's lateral driving intention corresponds to the evaluation problem of the hidden Markov model, that is, the problem of estimating the probability value. The core of this problem is to find the probability of the driver's lateral driving intention being a certain value under a certain traffic situation and driver state; The driver's lateral driving intention is expressed as Q, which is defined as follows: Q = -3, -2, -1, 0, 1, 2 respectively indicate that the driver's current lateral driving intention is to turn around, turn left, change lanes to the left, keep going straight, change lanes to the right, and turn right; The driver's head movement signal, eye movement signal, driving environment and driving operation signal transmitted by the vehicle CAN bus are input as observation variables, and after time discretization, the observation variable sequence O1, O2, ..., O t , the driver's lateral driving intention is used as the hidden state variable of the hidden Markov model, and the hidden state variable sequence Q1, Q2, ..., Q is obtained after time discretization. t ; The parameters of the hidden Markov model are defined as λ = (π, A, B), where π represents the initial state probability of the system, A represents the state transfer matrix, B represents the emission matrix, and the initial state probability π: the probability of each state of the model appearing at the initial moment, denoted as π = (π1, π2, ..., π t ), π i Represents the probability that the initial observed variable data of the model is Oi, and the state transition probability A: the probability of the model switching between various hidden states, recorded as the matrix A = [a ij ], where a ij It means that at any time t, if the hidden state variable data is Q i , then at the next moment the hidden state variable data is Q j The probability of the emission matrix B is the probability relationship between the model observation variables and the hidden state variables, recorded as matrix B = [b jk ], where b jk It means that at any time t, if the observed variable data is O j , then at the current moment the hidden state variable data is Q k probability; According to the parameters defined above, experiments are conducted to collect a sufficiently large data set, and then the data set is initialized and grouped according to the K-means clustering algorithm to obtain the initial parameter λ0 of the model; Step 2: Model training: Model training refers to repeated deep learning training based on the obtained data set and the principle of hidden Markov model to improve the accuracy of the model so that it can properly understand the driver's lateral driving intention; The main content of model training is: using the forward-backward algorithm to calculate the observed variable data used in the current training in the collected data set train The output probability P(O train |λ), apply the EM algorithm to update the model parameters λ, specifically including calculating the new state transfer matrix A and the new emission matrix B, repeating the training by replacing the observed variable data and the corresponding hidden state variable data, and updating P(O train |λ) is monitored, and when the value converges to a very small change or no longer changes, the training ends, and the hidden Markov model parameter λ obtained after the last training is end As the final hidden Markov model parameters, a hidden Markov model for understanding the actual driver's lateral driving intention is obtained; Link 3: Understanding the driver's lateral driving intention: After training, the hidden Markov model is used to understand the driver's lateral driving intention. The understanding process is the decoding problem solving process of the hidden Markov model. The decoding process uses the Viterbi algorithm to discretize the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus within a period of time to obtain the observed variable sequence O1, ..., O t Input the model and find the probability that the driver's lateral driving intention Q at the next moment is -3, -2, -1, 0, 1, and 2 respectively. The one with the largest probability is taken as the final output of the driver's lateral driving intention. Step 2: Understanding the longitudinal driving intention of the vehicle: The driver's longitudinal driving intention is inferred through the driver's head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus. The driver's longitudinal driving intention includes but is not limited to following the vehicle, accelerating, braking, stopping and holding; Step 3: Understanding other drivers’ driving intentions: Infer the driver’s other driving intentions through the driver’s head movement signals, eye movement signals, driving environment and driving operation signals transmitted by the vehicle CAN bus. The driver’s other driving intentions include but are not limited to hesitation and overtaking. The above steps 1, 2 and 3 are for understanding different aspects of the driver's driving intention, which are decoupled from each other and do not interfere with each other. They are judged in parallel during reasoning. The judgments of steps 1, 2 and 3 are similar. The methods of steps 2 and 3 are similar to those of step 1. Step 5: Generate human-machine hybrid enhanced perception map. The specific steps are as follows: Step 1: Modeling the driver’s perception target: In most cases, the driver can correctly control driving. Therefore, the first step for the machine and driver to achieve fusion perception should be to understand the driver’s perception target. The driver’s perception target modeling includes the following steps: Step 1: Modeling the driver's gaze direction: Model the driver's current gaze direction vector based on the detected head movement signal and eye movement signal. The gaze direction vector includes the position of the driver's eye center in the vehicle coordinate system (xe, ye, ze) and the unit vector of the gaze direction in the vehicle coordinate system (xd, yd, zd); Step 2: Driver’s Region of Interest (ROI) Judgment: When the driver’s gaze direction remains in the same or similar area for a period of time, the area can be considered as the driver’s region of interest, and the objects in the driver’s region of interest are the driver’s perception targets; Step 3: Modeling the driver's perception target: After the driver's perception target is confirmed, it needs to be displayed in the human-machine hybrid enhanced perception map in step 4. Therefore, it is necessary to establish a structure sequence to store the output target information of the perception map. In the human-machine hybrid enhanced perception map, the object's ID, relative position, relative speed and TTC are recorded and output in the human-machine hybrid enhanced perception map in step 4; Relative position refers to the shortest distance of an object relative to the edge of the vehicle, rather than the position of the object in the vehicle coordinate system, in order to adapt to the driver's perception habits. Relative speed is expressed as the sum of speeds, and the positive and negative speed values ​​are defined as follows: when the angle between the relative speed vector and the main vehicle speed vector is acute, it is positive, and when the angle is obtuse, it is negative; The driver's attention targets are analyzed based on the previously calculated driving situation, and flags are set to mark targets that seriously affect the current driving situation. A flag of 0 indicates a moderate impact; a flag of 1 indicates a large impact and requires special attention; a flag of 2 indicates a very large impact and an emergency, which also indicates that the current driving situation is judged to be dangerous; Step 2: Modeling the driver’s perception blind spots: The driver’s perception is limited. There are many blind spots that the driver cannot perceive during driving. The information of these blind spots will often become the key to avoiding accidents. The driver’s perception blind spots mainly include: Front blind spot: refers to the area in front of the hood, caused by the height of the vehicle body and the length of the front of the vehicle; Rear blind spot: refers to the area behind the vehicle that cannot be seen by the rearview mirror and reflector. When modeling, it is approximately considered that all areas behind the vehicle are treated as blind spots because they cannot be adequately observed; Rearview mirror blind spot: refers to the area on both sides of the vehicle that is included due to the limited field of view of the rearview mirror. The AB pillar blind spot is the area on both sides of the windshield that cannot be observed due to the obstruction of the AB pillars. If there is an object within the above-mentioned blind spot area, the driver will be informed as key information. When modeling, a spatial position model of the area covered by the blind spot is established based on the principle that light propagates in a straight line; Step 3: Machine perception map modeling: The machine perception map mainly includes the parts that the driver cannot actively perceive and require machine assistance, mainly including blind spot perception map modeling, driver inattention perception map modeling and demand perception map based on driver driving intention; blind spot perception map modeling refers to screening out all objects in the blind spot by comparing the spatial position of objects in the surrounding driving environment with the spatial position of the blind spot, recording the object's ID, relative position, relative speed and TTC information, and calculating the severity of the impact of the object on the current driving situation, and then sending it to the human-machine hybrid enhanced perception map in step 4. The driver inattention perception map refers to objects in the surrounding driving environment that are not in the blind spot or in the driver's area of ​​interest, but should be noticed by the driver. The screening of such objects is achieved by calculating the object's The impact severity mark of the current driving situation is completed. After calculation, the ID, relative position, relative speed, TTC and severity mark of the objects with marks greater than 0 are recorded and sent to the human-machine hybrid enhanced perception map in step 4, similar to the previous one. The demand perception map based on the driver's driving intention means that when the driver completes a driving operation, he often needs to perceive multiple targets. When the driver changes lanes, he needs to observe the vehicle in front of him and the vehicle behind him. In addition to the area of ​​interest described in step 1, he also needs to perceive objects at specific locations according to the driver's driving intention. The input of this link is the three types of driver driving intentions obtained in step 4. The perception results of the above-mentioned objects at specific locations are obtained by using a high-dimensional space search algorithm, and are presented to the driver through the HMI as output. Step 4: Modeling of human-machine hybrid enhanced perception map: The human-machine hybrid enhanced perception map is the output of the entire system and is provided to both the driver and the intelligent driving system for perception result output. The perception map provided to the driver mainly includes the quantitative data information of all objects perceived in steps 1, 2 and 3 in the fifth step that need to be displayed to the driver, the traffic light information ahead and the traffic sign information of the current driving road. The perception map provided to the intelligent driving system includes all the information of all objects in the surrounding driving environment, driving situation information, and the driver's driving intention information. The output links of the human-machine hybrid enhanced perception map to the driver are as follows: Link 1: Output of perception map to the driver: The ID, relative position, relative speed and TTC information of the driver's perceived target are projected directly on the front windshield through HUD head-up display technology, and displayed in the direction of the driver's perceived target; the same treatment is performed for the objects in the driver's negligent perception map identified in step 3 of step 5; if the object in the blind spot perception map identified in step 3 of step 5 exists, it will be prompted in the form of a warning at the bottom of the front windshield. In addition to the conventional ID, relative speed and TTC information, the warning content also needs to include the blind spot position to replace the position relative to the main vehicle. This is more intuitive and can more effectively reduce the driver's perception burden and achieve a good human-machine enhanced perception effect; For the above-mentioned atlas display, different colors are displayed according to the position of the severity mark of the impact of the object on the current driving situation in the perception atlas. For objects with a severity mark of 0, green is displayed, for objects with a severity mark of 1, orange is displayed, and for objects with a severity mark of 2, red is displayed; When displaying, if there are two or more objects that overlap, the severity flag of the impact of the object on the current driving situation is used to judge, and the object with a larger severity flag value is displayed first; if there are two or more objects that overlap and have the same severity flag value, they are sorted according to the relative distance, and the object with a closer relative distance is displayed first; In addition, traffic light signals and traffic sign signals are displayed on the top of the front windshield. When the driver violates traffic regulations, the top display will flash brightly to warn the driver. Link 2: Perception map output to human-machine co-driving intelligent driving system: As the perception layer of human-machine co-driving technology, the human-machine hybrid enhanced perception map needs to output relevant information of the driver, including the driver's driving intention. All information about objects in the surrounding driving environment and driving situation information serve as calculation input and output for subsequent planning, decision-making and control.