Vehicle automatic driving decision-making method and system based on multi-sensor fusion

Abstract

The invention is suitable for the technical field of intelligent driving, and provides a vehicle automatic driving decision-making method and system based on multi-sensor fusion, and the method comprises the following steps: collecting multi-sensor data in real time when a first automatic driving decision is executed; the multi-sensing data comprises in-vehicle image data, in-vehicle sound data and out-vehicle environment data; according to the in-vehicle image data and the in-vehicle sound data, the state abnormity degree of the driver is determined; whether the state abnormity degree of the driver is larger than a preset safety threshold value or not is judged in real time; if the state abnormity degree of the driver is greater than the preset safety threshold value, automatically switching to a second automatic driving decision; otherwise, the first automated driving decision is maintained. According to the method, the state of a driver is judged in real time through a multi-sensor fusion technology, and after the state of the driver is determined to be abnormal, an optimal parking point which comprehensively considers factors such as path distance, traffic flow safety and parking space size is selected and navigated to a vehicle through a multi-target optimization function.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving technology, and in particular relates to a vehicle autonomous driving decision-making method and system based on multi-sensor fusion. Background Technology

[0002] The so-called "autonomous driving" currently on the market actually falls under the category of "assisted driving" in most application scenarios. Its system design logic still takes the driver as the ultimate responsible party, requiring the driver to take over the vehicle in a timely manner when the system encounters a situation that it cannot handle.

[0003] However, this logic presents a fundamental safety hazard: when a vehicle is driven to its destination under the control of an autonomous driving system, if there is no human intervention throughout the journey, the system will default to maintaining the current driving decision. Once the driver falls into a state of deep fatigue, or even falls asleep or suddenly loses consciousness, the entire system is exposed to enormous risks. This is because if the autonomous driving system itself malfunctions, exceeds its designed operating range (such as a sudden sensor failure), or encounters unrecognizable sudden road conditions (such as a road collapse ahead or an illegal intrusion), the system will be stuck in a decision-making deadlock because it cannot obtain effective control from the driver. It will either blindly continue driving or execute inappropriate emergency measures (such as sudden braking at high speed), either of which can easily lead to a serious traffic accident.

[0004] While existing technologies involve driver status monitoring to some extent, they generally fail to deeply integrate the monitoring results with the autonomous driving decision-making chain. They lack a complete mechanism that can autonomously, intelligently, and safely take over and execute the globally optimal risk avoidance decision after confirming driver abnormality, which results in a serious gap in the safety assurance of the current system. Summary of the Invention

[0005] The purpose of this invention is to provide a vehicle autonomous driving decision-making method based on multi-sensor fusion, which aims to solve the above-mentioned technical problems.

[0006] This invention is implemented as follows: a vehicle autonomous driving decision-making method based on multi-sensor fusion, comprising the following steps:

[0007] When executing the first autonomous driving decision, multi-sensor data is collected in real time; the multi-sensor data includes in-vehicle image data, in-vehicle sound data, and external environment data.

[0008] The degree of abnormality of the driver's state is determined based on in-vehicle video data and in-vehicle audio data;

[0009] The system continuously assesses whether the driver's abnormal state exceeds a preset safety threshold. If the driver's abnormal state exceeds the preset safety threshold, it automatically switches to the second autonomous driving decision; otherwise, it maintains the first autonomous driving decision.

[0010] The second autonomous driving decision includes:

[0011] Based on external environmental data, identify multiple candidate parking spots;

[0012] Based on a preset multi-objective optimization function, a safe parking space is selected from the candidate parking points;

[0013] The system automatically controls the vehicle to drive to a safe parking space and performs a parking operation.

[0014] Furthermore, the steps for determining the degree of abnormality in the driver's state based on in-vehicle video data and in-vehicle audio data specifically include:

[0015] Extracting the driver's visual features from in-vehicle image data;

[0016] Extracting the driver's voice features from in-vehicle audio data;

[0017] Based on a pre-defined state assessment model, the driver's state abnormality is generated according to the driver's visual and vocal characteristics.

[0018] Furthermore, the visual features include eye features, lip features, head features, and posture features.

[0019] Furthermore, the state evaluation model is a multimodal fusion deep learning network, including a feature extraction encoder, a multimodal fusion module, and a decision output layer.

[0020] Furthermore, the feature extraction encoder includes a visual coding branch and a sound coding branch; the visual coding branch is based on a bidirectional long short-term memory network and generates a visual temporal feature vector based on visual features; the sound coding branch is based on a bidirectional long short-term memory network and generates a sound temporal feature vector based on sound features.

[0021] The multimodal fusion module performs weighted fusion of visual temporal feature vectors and audio temporal feature vectors based on an attention mechanism to obtain a fused feature vector;

[0022] The decision output layer will fuse feature vectors and perform nonlinear transformations, and output the state anomaly degree through an activation function.

[0023] Furthermore, the step of selecting a safe parking space from candidate parking spots based on a preset multi-objective optimization function specifically includes:

[0024] Multiple candidate parking navigation routes are generated based on the vehicle's current location and the location of each candidate parking spot;

[0025] Determine the path distance and traffic flow status for each candidate parking navigation route;

[0026] Determine the available spatial geometry for each candidate parking spot;

[0027] Based on the path distance and traffic flow status of the parking navigation route and the available spatial geometry of the parking spot, a multi-objective optimization function is determined, and the comprehensive safety efficiency index of each candidate parking spot is calculated.

[0028] The candidate parking spot with the highest comprehensive safety timeliness index was selected as the safe parking space.

[0029] Furthermore, the traffic flow status includes traffic flow density and vehicle interaction collision risk.

[0030] Another object of the present invention is to provide a vehicle autonomous driving decision-making system based on multi-sensor fusion, for implementing the above-mentioned vehicle autonomous driving decision-making method, comprising:

[0031] The multi-sensor data acquisition module is used to acquire multi-sensor data in real time when executing the first autonomous driving decision; the multi-sensor data includes in-vehicle image data, in-vehicle sound data, and external environment data.

[0032] The abnormality determination module is used to determine the abnormality of the driver's state based on in-vehicle video data and in-vehicle audio data.

[0033] The driver state judgment module is used to determine in real time whether the abnormality of the driver's state exceeds a preset safety threshold; if the abnormality of the driver's state exceeds the preset safety threshold, it automatically switches to the second autonomous driving decision; otherwise, it maintains the first autonomous driving decision.

[0034] The second autonomous driving decision module is used to execute the second autonomous driving decision;

[0035] The second autonomous driving decision module specifically includes:

[0036] The candidate parking spot identification unit is used to identify multiple candidate parking spots based on external environmental data.

[0037] The safe parking space selection unit is used to select a safe parking space from candidate parking points based on a preset multi-objective optimization function;

[0038] The safe parking execution unit is used to automatically control the vehicle to drive to a safe parking space and perform parking operations.

[0039] Furthermore, the state anomaly determination module specifically includes:

[0040] The visual feature extraction unit is used to extract the driver's visual features from in-vehicle image data.

[0041] The sound feature extraction unit is used to extract the driver's voice features from in-vehicle sound data;

[0042] The state assessment unit is used to generate the driver's state abnormality based on the driver's visual and auditory characteristics, according to a preset state assessment model.

[0043] Furthermore, the safe parking space selection unit specifically includes:

[0044] The navigation route generation sub-unit is used to generate multiple candidate parking navigation routes based on the vehicle's current location and the location of each candidate parking point.

[0045] The navigation route analysis sub-unit is used to determine the path distance and traffic flow status of each candidate parking navigation route;

[0046] The parking spot analysis sub-unit is used to determine the available spatial geometry of each candidate parking spot.

[0047] The multi-objective optimization sub-unit is used to determine the multi-objective optimization function based on the path distance of the parking navigation route, traffic flow status, and available spatial geometry of the parking point, and to calculate the comprehensive safety efficiency index for each candidate parking point;

[0048] The parking spot determination sub-unit is used to select the candidate parking spot with the highest comprehensive safety timeliness index as the safe parking space.

[0049] This invention provides a vehicle autonomous driving decision-making method based on multi-sensor fusion. It uses multi-sensor fusion technology to determine the driver's state in real time, and upon confirming an abnormal driver state, it can proactively and intelligently switch to another independent autonomous driving decision. Under this decision, a multi-objective optimization function selects and navigates the vehicle to an optimal parking point that comprehensively considers factors such as path distance, traffic flow safety, and parking space size. This achieves an efficient and smooth decision-making switch from normal autonomous driving to safe parking. This not only effectively avoids traffic accidents caused by abnormal driver states but also completely eliminates the dangers that may arise from the autonomous driving system waiting for takeover or performing improper emergency operations, as well as the secondary dangers that may result from unreasonable emergency parking. It dynamically balances driving safety and road traffic efficiency, greatly improving the safety redundancy and reliability of the autonomous driving system. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a vehicle autonomous driving decision-making method based on multi-sensor fusion, provided as an embodiment of the present invention.

[0051] Figure 2This is a flowchart illustrating step S200 in a vehicle autonomous driving decision-making method based on multi-sensor fusion, provided in an embodiment of the present invention.

[0052] Figure 3 This is a flowchart illustrating step S420 in a vehicle autonomous driving decision-making method based on multi-sensor fusion, provided in an embodiment of the present invention.

[0053] Figure 4 This is a schematic diagram of a vehicle autonomous driving decision-making system based on multi-sensor fusion, provided as an embodiment of the present invention.

[0054] Figure 5 This is a schematic diagram of the structure of the second autonomous driving decision module provided in an embodiment of the present invention.

[0055] Figure 6 This is a schematic diagram of the state anomaly determination module provided in an embodiment of the present invention.

[0056] Figure 7 This is a schematic diagram of the structure of the safe parking space selection unit provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0058] like Figure 1 As shown, in one embodiment of the present invention, a vehicle autonomous driving decision-making method based on multi-sensor fusion is provided, including the following steps:

[0059] S100. When executing the first autonomous driving decision, multi-sensor data is collected in real time; the multi-sensor data includes in-vehicle image data, in-vehicle sound data, and external environment data.

[0060] S200: Determine the degree of abnormality of the driver's state based on in-vehicle image data and in-vehicle sound data;

[0061] S300: Real-time determination of whether the driver's state abnormality exceeds a preset safety threshold; if the driver's state abnormality exceeds the preset safety threshold, automatically switch to the second autonomous driving decision; otherwise, maintain the first autonomous driving decision.

[0062] The second autonomous driving decision includes:

[0063] S410: Identify multiple candidate parking spots based on external environmental data;

[0064] S420. Based on a preset multi-objective optimization function, select a safe parking space from the candidate parking points;

[0065] S430: Automatically controls the vehicle to drive to a safe parking space and performs a parking operation.

[0066] It should be noted that the first autonomous driving decision is the default decision of the autonomous driving system, which is usually the L2 / L3 level autonomous driving of existing technologies.

[0067] In practical applications, in-vehicle image data is typically collected by near-infrared cameras deployed behind the steering wheel or above the cockpit. Near-infrared light sources ensure clear imaging even at night or in environments with sudden changes in light, and the sampling rate is generally no less than 30fps to capture subtle physiological dynamics. In-vehicle audio data is collected via a directional microphone array, which uses beamforming technology to focus on the driver's voice source, effectively suppressing background interference such as engine noise, wind noise, and passenger conversations. External environmental data serves as the input for the first autonomous driving decision and forms the basis for the second autonomous driving decision. External environmental data is collected by sensors such as forward / surround-view cameras, millimeter-wave radar, and LiDAR, and fused with V2X data to construct a real-time 360-degree environmental perception of the vehicle's surroundings, including but not limited to: drivable areas, traffic participants, traffic signs, traffic lights, and potential parking spots (such as standard parking spaces, highway emergency lanes, shoulders, service area exit buffer zones, and road merging areas).

[0068] like Figure 2 As shown, in a preferred embodiment of the present invention, the step of determining the degree of abnormality of the driver's state based on in-vehicle image data and in-vehicle sound data, i.e., step S200, specifically includes:

[0069] S210. Extract the driver's visual features from in-vehicle image data;

[0070] S220. Extract the driver's voice characteristics from in-vehicle sound data;

[0071] S230. Based on the preset state assessment model, the driver's state abnormality is generated according to the driver's visual and auditory characteristics.

[0072] Specifically, the visual features include eye features, lip features, head features, and posture features. Lightweight convolutional neural networks (such as MobileNet-V3) can be used to process the video stream from in-vehicle image data in real time, outputting eye features such as eyelid closure degree, blink frequency, blink duration, and pupil diameter change rate. Eyelid closure degree is a key feature, which can initially determine whether the driver is deeply fatigued or asleep. It should be noted that if the driver only closes their eyes briefly but remains conscious, it cannot be determined that the driver is deeply fatigued or asleep. Therefore, other features need to be integrated to assess the driver's state. It is worth noting that eyelid closure degree can be defined as the percentage of time within a specific time window (e.g., 1-5 minutes) during which the eyelids cover more than 80% of the pupil's geometric size (i.e., greater than 80% of the pupil's geometric size when fully open). In other words, the driver's state is assessed by analyzing whether the driver is in a closed-eye state for an extended period.

[0073] Lip features can be used to locate the lip contour through facial landmark detection models (such as MediaPipe Face Mesh), calculate the degree of lip opening and the frequency and duration of yawning; lip features can be combined with in-vehicle sound data to determine whether the driver is talking or drinking or eating.

[0074] Head features can be used to calculate the Euler angles (pitch, yaw, roll) of the head relative to the torso based on key points, and analyze its motion trajectory to identify abnormal head drooping, continuous yaw and other behaviors.

[0075] Postural features can be extracted using human posture estimation models (such as OpenPose) to extract key points of the upper body skeleton and analyze the tilt of the shoulders and the degree of body slump, which are typical postures of coma or deep sleep.

[0076] Furthermore, step S220 specifically includes: first, preprocessing the audio signal (noise reduction, framing), then extracting Mel-frequency cepstral coefficients (MFCCs) as low-level features; next, using a recurrent neural network (RNN / LSTM) to model temporal dependencies and extract high-level features, such as:

[0077] Sound source location: Determine if the sound is coming from the driver and avoid interference from other sound sources;

[0078] Voice activity detection failed: No voice signal for an extended period of time;

[0079] Speech ambiguity: unclear or intermittent pronunciation can be quantified by the variance of the feature vector output by LSTM;

[0080] Abnormal groans or breathing sounds: as an indicator of sudden illness.

[0081] In a preferred embodiment of the present invention, the state evaluation model is a multimodal fusion deep learning network, including a feature extraction encoder, a multimodal fusion module, and a decision output layer.

[0082] Specifically, the feature extraction encoder includes a visual encoding branch and a sound encoding branch; the visual encoding branch is based on a bidirectional long short-term memory network and generates a visual temporal feature vector based on visual features; the sound encoding branch is based on a bidirectional long short-term memory network and generates a sound temporal feature vector based on sound features; the multimodal fusion module performs weighted fusion of the visual temporal feature vector and the sound temporal feature vector based on an attention mechanism to obtain a fused feature vector; the decision output layer performs a nonlinear transformation on the fused feature vector and outputs the state anomaly degree through an activation function.

[0083] In practical applications, the visual branch specifically includes: extracting the driver's eye features, lip features, head features, and posture features from in-vehicle image data according to the above method; this step removes irrelevant information such as image background and focuses on abstract features related to the driver's state; inputting the eye features, lip features, head features, and posture features into a bidirectional long short-term memory network in chronological order, which can learn contextual information to effectively capture dynamic evolution processes such as "the duration of eye closure is getting longer and longer" and "the head begins to tilt forward continuously"; the output of the last time step of the bidirectional long short-term memory network or the pooled sequence output is used as the final visual temporal feature vector.

[0084] The sound branch specifically includes: extracting the driver's voice features from the in-vehicle sound data according to the above method; inputting the sound features into another bidirectional long short-term memory network in chronological order; this bidirectional long short-term memory network can learn the temporal patterns of sound, such as the entire process of speech from clear to unclear, from coherent to intermittent, and then to silence, and its final output is used as the sound temporal feature vector.

[0085] The multimodal fusion module employs weighted fusion based on an attention mechanism. In practical applications, when the driver begins to speak, vocal features dominate; when the driver is silent but nodding off, visual features such as eye features, lip features, head features, and posture features dominate. Furthermore, at night, due to lighting conditions, the confidence level of visual features is low, so vocal features should be prioritized. When the driver's face is obscured by sunglasses or other coverings, the confidence level of visual features is significantly reduced, so vocal features should be prioritized. When there is a lot of noise inside the vehicle (such as loud music), the confidence level of vocal features is significantly reduced, so visual features should be prioritized. To address this, the embodiments of this invention employ an attention mechanism to dynamically adjust these parameters, thereby improving the accuracy of state assessment.

[0086] Specifically, the implementation method of the multimodal fusion module is as follows:

[0087] First, we first combine the two feature vectors (visual temporal feature vectors). and sound temporal feature vector ) Generate query vectors through linear transformation. Two key vectors (visual feature key vectors) and sound feature key vector ) and two value vectors (visual feature value vectors) and sound feature vector ), as detailed below:

[0088] ;

[0089] , ;

[0090] , ;

[0091] In the formula, Indicates a splicing operation; All of these are learnable parameters.

[0092] Next, the similarity between the query vector and each modality key vector (visual feature key vector and audio feature key vector) is calculated (by dot product), and then normalized to weights using the Softmax function to obtain the attention weights for visual features. Attention weights for sound features The details are as follows:

[0093] ;

[0094] ;

[0095] It should be noted that the attention weights of visual features and sound features here represent the relative importance of visual and sound features to the final decision at the current moment.

[0096] Then, by weighting and summing the value vectors of the visual features and the audio features using the attention weights mentioned above, the final fused feature vector can be obtained. ;

[0097] ;

[0098] The decision output layer performs a nonlinear transformation on the aforementioned fused feature vector through one or more fully connected layers, and finally maps the output through a scalar between 0 and 1 using a sigmoid activation function. This scalar value represents the driver's state abnormality. A state abnormality value close to 0 indicates that the driver is in a normal state and is attentive; a state abnormality value close to 1 indicates that the driver is in a highly abnormal state (such as deep sleep or coma).

[0099] In practical applications, the training method for the aforementioned state assessment model is as follows: First, under strict adherence to privacy regulations, a large amount of real-world and simulated driving data is collected. The data must cover: multiple states (normal, distracted, mild fatigue, severe fatigue, asleep, and simulated sudden illness scenarios, etc.); multiple environmental conditions (different lighting conditions, different weather conditions); and multiple drivers (different ages, genders, skin colors, whether they wear glasses, etc.). Next, labeling is performed: the data is labeled by experts according to strict protocols, providing a continuous state anomaly score as a supervision signal. Then, mean squared error loss is used as the loss function, with the goal of making the model's predicted state anomaly score as close as possible to the expert-labeled score.

[0100] In this embodiment of the invention, by judging in real time whether the abnormality of the driver's state is greater than a preset safety threshold, it is possible to accurately determine whether the driver is in an abnormal state such as deep fatigue, falling asleep or unconsciousness. It should be noted that the safety threshold can be dynamically adjusted according to vehicle speed, road type (highway / city), time and driver's historical behavior data, and is generally set to 0.8-0.9.

[0101] In step S410, the external environment data can be fused with data collected by sensors such as forward / surround view cameras, millimeter-wave radar, and lidar, and V2X data to construct a 360-degree environmental perception result around the vehicle in real time. Using the external environment perception result, a lightweight semantic segmentation model (such as DeepLabV3+) is used to process the real-time image or LiDAR point cloud to identify all areas that meet the basic geometric conditions as candidate parking points, such as: standard legal parking spaces on the roadside, emergency lanes on highways, road shoulders, service area exit buffer zones, and road diversion areas.

[0102] like Figure 3 As shown, in a preferred embodiment of the present invention, the step of selecting a safe parking space from candidate parking spots based on a preset multi-objective optimization function, i.e., step S420, specifically includes:

[0103] S421. Generate multiple candidate parking navigation routes based on the vehicle's current location and the location of each candidate parking point;

[0104] S422. Determine the path distance and traffic flow status for each candidate parking navigation route; determine the timeliness factor based on the path distance and the safety factor based on the traffic flow status; the calculation formula for the timeliness factor is as follows:

[0105] ;

[0106] In the formula, T i d is the time factor for the i-th candidate parking point; i d is the path distance of the i-th candidate parking navigation route; max The maximum allowed path indicates that if the parking spot is too far from the current vehicle location, it will not be given priority.

[0107] The traffic flow status includes traffic flow density and vehicle collision risk; the safety factor is calculated using the following formula:

[0108] ;

[0109] , ;

[0110] ;

[0111] In the formula, S i R is the safety factor for the i-th candidate parking point; i P represents the density risk index of the i-th candidate parking spot. i Let be the collision risk index for the i-th candidate parking point; and The weighting coefficients are given, and the sum of the two is 1. These coefficients can be determined based on the specific circumstances. The value typically ranges from 0.3 to 0.5. The value is generally between 0.5 and 0.7; N i N represents the current number of vehicles (including the vehicle itself) for the i-th candidate parking navigation route; maxi The maximum allowed number of vehicles for the i-th candidate parking navigation route; TTC i The average collision time between the vehicle and surrounding adjacent vehicles (especially vehicles in front and to the side in the same lane) in the i-th candidate parking navigation route; the collision time refers to the time required for two vehicles to maintain their current state of motion until a collision occurs, and is used to assess vehicle safety risk; n is the total number of surrounding adjacent vehicles. Let be the distance between the vehicle itself and the j-th neighboring vehicle in the i-th candidate parking navigation route; Let be the relative speed between the vehicle itself and the j-th neighboring vehicle in the i-th candidate parking navigation route.

[0112] S423. Determine the available space geometry for each candidate parking spot; available space geometry refers to the geometry (including length and width) of a valid, continuous space without obstacles that allows for legal parking; determine the tolerance factor based on the available space geometry; the formula for calculating the tolerance factor is as follows:

[0113] ;

[0114] In the formula, E i G is the tolerance factor for the i-th candidate parking point; i G represents the available spatial geometry (length or width) of the i-th candidate parking point; min This represents the lower limit of the available space geometry at the parking spot, generally the same as the vehicle's own geometry (length or width); G max This represents the upper limit of the available space geometry of the parking spot, generally 1.5-2 times the geometric dimensions (length or width) of the vehicle itself. It should be noted that if both length and width are considered, the tolerance factor for each dimension needs to be calculated separately, and then the geometric mean of the two factors is taken to obtain the final tolerance factor.

[0115] S424. Based on the path distance and traffic flow status of the parking navigation route, as well as the available spatial geometry of the parking spots, i.e., based on the aforementioned timeliness factor, safety factor, and fault tolerance factor, determine the multi-objective optimization function (the objective of which is to find a parking spot that is optimal in terms of safety and efficiency), and calculate the comprehensive safety and timeliness index for each candidate parking spot; the multi-objective optimization function is as follows:

[0116] ;

[0117] In the formula, Z i is the comprehensive safety timeliness index of the i-th candidate parking point; w1, w2 and w3 are all weighting coefficients, the sum of which is 1, and can be determined according to the actual situation; generally, w1 takes the value of 0.1-0.3, w2 takes the value of 0.4-0.8, and w3 takes the value of 0.1-0.3.

[0118] S425. Select the candidate parking spot with the highest comprehensive safety timeliness index as the safe parking space.

[0119] In step 430, in practical applications, once a target safe parking space is selected, the system will invoke a motion planning algorithm (such as the existing technology of path planning combined with speed planning based on Lattice Planner) to generate a smooth and safe trajectory. This trajectory is then tracked, while real-time obstacle avoidance is integrated. The entire parking process will activate hazard lights and broadcast the vehicle's emergency status to surrounding vehicles via V2X technology until the vehicle comes to a complete stop within the safe parking space. Furthermore, during the execution of the second autonomous driving decision, a multi-level, progressively escalating wake-up procedure can be initiated, with the core principles of effectiveness and non-startleness, such as voice wake-up, visual flashing wake-up, and vibration / tactile wake-up. When the driver's abnormal state significantly and continuously drops below a safe threshold, the driver can be alerted that they are currently in the second autonomous driving decision phase and allowed to decide whether to switch back to the first autonomous driving decision phase or switch to manual driving mode.

[0120] like Figure 4 As shown, in another embodiment of the present invention, a vehicle autonomous driving decision-making system based on multi-sensor fusion is also provided to implement the above-described vehicle autonomous driving decision-making method, comprising:

[0121] The multi-sensor data acquisition module 10 is used to acquire multi-sensor data in real time when executing the first autonomous driving decision; the multi-sensor data includes in-vehicle image data, in-vehicle sound data and external environment data;

[0122] The state anomaly determination module 20 is used to determine the driver's state anomaly level based on in-vehicle image data and in-vehicle sound data.

[0123] The driver state judgment module 30 is used to judge in real time whether the abnormality of the driver's state is greater than a preset safety threshold; if the abnormality of the driver's state is greater than the preset safety threshold, it automatically switches to the second autonomous driving decision; otherwise, it maintains the first autonomous driving decision.

[0124] The second autonomous driving decision module 40 is used to execute the second autonomous driving decision;

[0125] like Figure 5 As shown, the second autonomous driving decision module 40 specifically includes:

[0126] The candidate parking spot identification unit 41 is used to identify multiple candidate parking spots based on external environment data.

[0127] The safe parking space selection unit 42 is used to select a safe parking space from candidate parking points based on a preset multi-objective optimization function;

[0128] The safe parking execution unit 43 is used to automatically control the vehicle to drive to a safe parking space and perform a parking operation.

[0129] like Figure 6 As shown, in a preferred embodiment of the present invention, the state anomaly determination module 20 specifically includes:

[0130] The visual feature extraction unit 21 is used to extract the driver's visual features from the in-vehicle image data.

[0131] The sound feature extraction unit 22 is used to extract the driver's voice features from the in-vehicle sound data;

[0132] The state assessment unit 23 is used to generate the driver's state abnormality based on the driver's visual and auditory characteristics according to the preset state assessment model.

[0133] like Figure 7 As shown, in a preferred embodiment of the present invention, the safe parking space selection unit 42 specifically includes:

[0134] The navigation route generation sub-unit 421 is used to generate multiple candidate parking navigation routes based on the current location of the vehicle and the location of each candidate parking point.

[0135] The navigation route analysis sub-unit 422 is used to determine the path distance and traffic flow status of each candidate parking navigation route.

[0136] Parking point analysis sub-unit 423 is used to determine the available spatial geometry of each candidate parking point;

[0137] The multi-objective optimization sub-unit 424 is used to determine the multi-objective optimization function based on the path distance and traffic flow status of the parking navigation route and the available spatial geometry of the parking point, and to calculate the comprehensive safety timeliness index of each candidate parking point.

[0138] The parking spot determination sub-unit 425 is used to select the candidate parking spot with the highest comprehensive safety timeliness index as the safe parking space.

[0139] It should be noted that each of the above modules or units can be implemented as a computer program, which can run on a computer device. The computer device's memory can store the computer program that makes up each module, enabling the processor to execute each step of the above method.

[0140] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0141] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0142] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A vehicle automatic driving decision-making method based on multi-sensor fusion, characterized in that, The method comprises the following steps: collecting multi-sensor data in real time when executing a first automatic driving decision; the multi-sensor data comprises in-vehicle image data, in-vehicle sound data and out-of-vehicle environment data; determining a state abnormality degree of the driver according to the in-vehicle image data and the in-vehicle sound data; determining whether the state abnormality degree of the driver is greater than a preset safety threshold in real time; if the state abnormality degree of the driver is greater than the preset safety threshold, automatically switching to a second automatic driving decision; otherwise, maintaining the first automatic driving decision; the second automatic driving decision comprises: identifying a plurality of candidate parking points according to the out-of-vehicle environment data; selecting a safe parking space from the candidate parking points based on a preset multi-objective optimization function; automatically controlling the vehicle to travel to the safe parking space and perform a parking operation. 2.The method of claim 1, wherein, The step of determining the state abnormality degree of the driver according to the in-vehicle image data and the in-vehicle sound data specifically comprises: extracting visual features of the driver from the in-vehicle image data; extracting sound features of the driver from the in-vehicle sound data; generating the state abnormality degree of the driver based on a preset state evaluation model according to the visual features and the sound features of the driver. 3.The method of claim 2, wherein, The visual features comprise eye features, lip features, head features and posture features. 4.The method of claim 2, wherein, The state evaluation model is a multi-modal fusion deep learning network comprising a feature extraction encoder, a multi-modal fusion module and a decision output layer.

5. The multi-sensor fusion based vehicle automatic driving decision method according to claim 4, characterized in that, The feature extraction encoder comprises a visual coding branch and a sound coding branch; the visual coding branch generates visual time sequence feature vectors based on a bidirectional long short-term memory network and according to the visual features; the sound coding branch generates sound time sequence feature vectors based on a bidirectional long short-term memory network and according to the sound features; The multi-modal fusion module performs weighted fusion on the visual time sequence feature vectors and the sound time sequence feature vectors based on an attention mechanism to obtain a fusion feature vector; The decision output layer performs nonlinear transformation on the fusion feature vector and outputs the state abnormality degree through an activation function.

6. The multi-sensor fusion based vehicle automatic driving decision method according to claim 1, characterized in that, The step of selecting a safe parking space from the candidate parking points based on a preset multi-objective optimization function specifically comprises: generating a plurality of candidate parking navigation routes according to the current position of the vehicle and the position of each candidate parking point respectively; determining the path distance and the traffic flow state of each candidate parking navigation route respectively; determining the available space geometric size of each candidate parking point; determining the multi-objective optimization function according to the path distance and the traffic flow state of the parking navigation route and the available space geometric size of the parking point, and calculating the comprehensive safety and timeliness index of each candidate parking point; selecting the candidate parking point with the maximum comprehensive safety and timeliness index as the safe parking space.

7. The multi-sensor fusion based vehicle automatic driving decision method according to claim 6, characterized in that, The traffic flow state comprises traffic flow density and vehicle interaction collision risk.

8. A multi-sensor fusion based vehicle automatic driving decision system for implementing the vehicle automatic driving decision method of any one of claims 1-7, characterized in that, The method comprises: a multi-sensor data collection module for collecting multi-sensor data in real time when executing a first automatic driving decision; the multi-sensor data comprises in-vehicle image data, in-vehicle sound data and out-of-vehicle environment data; a state abnormality degree determination module for determining a state abnormality degree of the driver according to the in-vehicle image data and the in-vehicle sound data; The driver state judgment module is configured to determine whether the state abnormality degree of the driver is greater than a preset safety threshold in real time. If the state abnormality degree of the driver is greater than the preset safety threshold, the first automatic driving decision is automatically switched to a second automatic driving decision; otherwise, the first automatic driving decision is maintained. The second automatic driving decision module is configured to execute the second automatic driving decision. The second automatic driving decision module specifically includes: A candidate parking point identification unit configured to identify a plurality of candidate parking points according to off-vehicle environment data. A safe parking space selection unit configured to select a safe parking space from the candidate parking points based on a preset multi-objective optimization function. A safe parking execution unit configured to automatically control the vehicle to travel to the safe parking space and perform a parking operation. 9.The multi-sensor fusion based vehicle automatic driving decision system according to claim 8, wherein, The state abnormality degree determination module specifically includes: A visual feature extraction unit configured to extract visual features of the driver from in-vehicle image data. An acoustic feature extraction unit configured to extract acoustic features of the driver from in-vehicle sound data. A state evaluation unit configured to generate a state abnormality degree of the driver based on a preset state evaluation model and the visual features and the acoustic features of the driver.

10. The multi-sensor fusion based vehicle automatic driving decision system according to claim 8, wherein, The safe parking space selection unit specifically includes: A navigation route generation subunit configured to generate a plurality of candidate parking navigation routes according to the current position of the vehicle and the position of each candidate parking point. A navigation route analysis subunit configured to determine the path distance and the traffic flow state of each candidate parking navigation route. A parking point analysis subunit configured to determine the available space geometric dimensions of each candidate parking point. A multi-objective optimization subunit configured to determine a multi-objective optimization function according to the path distance and the traffic flow state of the parking navigation route and the available space geometric dimensions of the parking point, and calculate a comprehensive safety and timeliness index of each candidate parking point. A parking point determination subunit configured to select the candidate parking point with the maximum comprehensive safety and timeliness index as the safe parking space.

Images