Assistant driving method, device, electronic device and storage medium

Abstract

The present application discloses an assisted driving method, device, electronic device and storage medium, relating to the technical field of assisted driving. The method includes: obtaining perception information, where the perception information includes vehicle environment information and vehicle state information; making a driving mode decision according to the perception information to obtain a first decision result; collecting electroencephalogram signals of the driver to obtain electroencephalogram information; performing classification processing on the electroencephalogram information to obtain a second decision result, where the second decision result is used to represent the driving intention of the driver; performing fusion analysis on the first decision result and the second decision result to obtain a fusion analysis result, and performing assisted driving control on the vehicle according to the fusion analysis result. The present application controls the vehicle by combining the decision results made based on the perception information and the electroencephalogram signals of the driver, improves the synchronization and safety of intelligent driving, and enhances the user experience.

Description

Technical Field

[0001] The present application relates to the field of assisted driving technology, and in particular, to an assisted driving method, device, electronic device, and storage medium. Background Art

[0002] At present, road traffic injuries have become one of the important causes of death in the population. Approximately 95% of various motor vehicle accidents are caused by improper driver operation to a certain extent, and traffic accidents completely induced by inappropriate driving behaviors of drivers account for three-quarters of the total number of accidents. Therefore, it is very necessary to analyze driver behavior and driver state.

[0003] In order to improve driving safety, existing research related to assisted driving mainly falls into two categories: one is to use in-vehicle sensors (such as cameras, infrared detection, etc.) to obtain obstacle information in front of the vehicle and determine whether there are potential risk factors, so as to take some control measures to avoid the vehicle colliding with the obstacle in front. The disadvantage of this type of technology is that external devices are easily affected by weather and external environmental changes, and the detection effect is not good; the other is to use external devices (such as cameras for face recognition, infrared devices for induction, etc.) to predict and analyze the driving behavior or intention of the driver, and adjust the driving strategy according to the analysis results. However, due to the latency of the driver's driving behavior, the detected signal is difficult to timely reflect the change of the driver's state. In addition, the detection signal and the driver behavior signal are analyzed separately, and a good synchronous detection effect cannot be achieved. Summary of the Invention

[0004] The purpose of the present invention is to provide an assisted driving method, device, electronic device, and storage medium, aiming to improve the synchronization and safety of assisted driving.

[0005] An embodiment of the present application provides an assisted driving method, including:

[0006] Obtain perception information, where the perception information includes vehicle environment information and vehicle state information;

[0007] Make a driving mode decision according to the perception information to obtain a first decision result;

[0008] Collect electroencephalogram signals of the driver to obtain electroencephalogram information;

[0009] Classify the electroencephalogram information to obtain a second decision result, where the second decision result is used to represent the driving intention of the driver;

[0010] Fusion analyze the first decision result and the second decision result to obtain a fusion analysis result, and perform assisted driving control on the vehicle according to the fusion analysis result.

[0011] In some embodiments of the present application, making a driving mode decision based on the perception information to obtain a first decision result includes:

[0012] Performing feature extraction processing on the perception information to obtain an extraction result;

[0013] Determining the driving area of the vehicle based on the extraction result, and setting the driving route of the vehicle according to the driving area;

[0014] Performing a state evaluation process on the vehicle according to the driving area and the driving route to obtain a state evaluation result, and selecting a driving mode according to the state evaluation result, where the driving mode includes any one of uniform speed driving, following vehicle driving, and emergency braking;

[0015] Outputting a control instruction based on the selected driving mode as the first decision result.

[0016] In some embodiments of the present application, outputting a control instruction based on the selected driving mode as the first decision result includes:

[0017] When the driving mode is selected as following vehicle driving, dividing the current following vehicle area according to the driving area to obtain a following vehicle driving area and a collision avoidance area;

[0018] Judging the area where the vehicle is currently located;

[0019] If the vehicle is in the following vehicle driving area, adjusting the target speed of the vehicle during driving according to the actual distance between the vehicle and the vehicle in front, so that the target speed maintains a negatively correlated relationship with the actual distance between the vehicle and the vehicle in front;

[0020] If the vehicle is in the collision avoidance area, adjusting the target speed to zero.

[0021] In some embodiments of the present application, classifying the brain wave information to obtain a second decision result includes:

[0022] Performing preprocessing on the brain wave information to obtain preprocessed brain wave information;

[0023] Using a trained convolutional neural network to perform classification testing on the preprocessed brain wave information to obtain a classification result;

[0024] Identifying the driving intention of the driver based on the classification result to obtain an identification result, and converting the identification result into a control instruction as the second decision result.

[0025] In some embodiments of the present application, using a trained convolutional neural network to perform classification testing on the preprocessed brain wave information to obtain a classification result includes:

[0026] Perform tagging processing on the preprocessed electroencephalogram information to obtain tagged electroencephalogram information;

[0027] Input the tagged electroencephalogram information in the form of frames into the trained convolutional neural network for classification testing, and rearrange the output results of the trained convolutional neural network according to the tags of the tagged electroencephalogram information as the classification results.

[0028] In some embodiments of the present application, the fusing and analyzing the first decision result and the second decision result to obtain a fused analysis result, and performing assisted driving control on the vehicle according to the fused analysis result includes:

[0029] Identify the braking intention of the driver in the second decision result;

[0030] If the second decision result includes a braking intention, use the second decision result as the fused analysis result;

[0031] If the second decision result does not include a braking intention, use the first decision result as the fused analysis result;

[0032] Based on the fused analysis result, use a preset control output model to calculate a control output amount, and perform assisted driving control on the vehicle according to the control output amount.

[0033] In some embodiments of the present application, based on the fused analysis result, using a preset control output model to calculate a control output amount, and performing assisted driving control on the vehicle according to the control output amount includes:

[0034] Construct a control output model based on the target speed, and the control output model is:

[0035]

[0036] where U t represents the control output amount, e(t)=v d -v a , v d represents the target speed, v a represents the current speed, K P represents the proportional amplification factor, K I represents the integral time constant, K D represents the differential time constant;

[0037] Perform discretization processing on the control output model to obtain a discrete control output model, and the discrete control output model is:

[0038]

[0039] Among them, T is the instruction cycle of the controller, and e j represents the speed deviation at time j, and e k represents the speed deviation at time k, and e k-1 represents the speed deviation at time k - 1;

[0040] Calculate the control output quantity using the discrete control quantity output model, and perform assisted driving control on the vehicle through the calculated control output quantity.

[0041] An embodiment of the present application provides an assisted driving device, and the device includes:

[0042] A sensing module, configured to obtain sensing information, where the sensing information includes vehicle environment information and vehicle state information;

[0043] A first decision-making module, configured to make a driving mode decision according to the sensing information to obtain a first decision result;

[0044] An electroencephalogram acquisition module, configured to acquire electroencephalogram signals of the driver to obtain electroencephalogram information;

[0045] A second decision-making module, configured to perform classification processing on the electroencephalogram information to obtain a second decision result, where the second decision result is used to represent the driving intention of the driver;

[0046] An analysis module, configured to perform fusion analysis on the first decision result and the second decision result to obtain a fusion analysis result, and perform assisted driving control on the vehicle according to the fusion analysis result.

[0047] An embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes any one of the assisted driving methods provided by the embodiments of the present application.

[0048] An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to perform any one of the assisted driving methods provided by the embodiments of the present application.

[0049] The beneficial effects of the present application: On the one hand, by obtaining sensing information to determine the vehicle environment and vehicle state, and thus making a decision result for the first type of assisted control; on the other hand, by collecting the electroencephalogram signals of the driver to determine the driving intention of the driver, and thus making a decision result for the second type of assisted control. Furthermore, by combining the decision results of the two types of assisted control to control the vehicle, the controlled vehicle makes a response under the combined influence of objective conditions and the subjective conditions of the driver, improving the safety of intelligent driving and enhancing the user experience. Brief Description of the Drawings

[0050] Figure 1 is a schematic diagram of the scenario of the assisted driving method provided by an embodiment of the present application.

[0051] Figure 2 is a schematic flowchart of an assisted driving method shown according to an exemplary embodiment.

[0052] Figure 3 is shown according to an exemplary embodiment Figure 2 a schematic flowchart of step S202 in

[0053] Figure 4 is shown according to an exemplary embodiment Figure 3 a schematic flowchart of step S304 in

[0054] Figure 5 is shown according to an exemplary embodiment Figure 2 a schematic flowchart of step S204 in

[0055] Figure 6 is shown according to an exemplary embodiment Figure 2 a schematic flowchart of step S205 in

[0056] Figure 7 is a schematic structural diagram of the assisted driving device provided by an embodiment of the present application.

[0057] Figure 8 is a schematic hardware structure diagram of the electronic device provided by an embodiment of the present application. Detailed Description of the Embodiments

[0058] To make the objectives, technical solutions and advantages of the present application clearer, the following will further describe in detail the embodiments of the present application with reference to the accompanying drawings.

[0059] First, the terms related to the embodiments of the present application will be introduced.

[0060] 1) Artificial Intelligence (AI)

[0061] Artificial intelligence is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, a theory, method, technology and application system that perceives the environment, acquires knowledge and uses the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0062] Artificial intelligence technology is a comprehensive discipline that involves a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0063] 2) Machine Learning (ML)

[0064] Machine learning is an interdisciplinary field that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning from demonstration.

[0065] Please refer to Figure 1 , which shows a schematic diagram of a computer system provided by an exemplary embodiment of the present application. The computer system includes a terminal 110 and a server 120. Among them, data communication is carried out between the terminal 110 and the server 120 through a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network, a metropolitan area network, and a wide area network.

[0066] An application program with electroencephalogram signal processing function is installed in the terminal 110. The application program can be a virtual reality application program, an assisted driving application program, or an artificial intelligence (AI) application program with electroencephalogram signal processing function. The embodiments of the present application do not limit this.

[0067] Optionally, the terminal 110 can be a terminal device with a brain-computer interface, and the brain-computer interface can obtain the electroencephalogram signal of the driver's head through electrodes; or the computer device has a data transmission interface, and the data transmission interface is used to receive the electroencephalogram signal collected by a data acquisition device with a brain-computer interface.

[0068] Optionally, the terminal 110 can be a mobile terminal such as a vehicle-mounted computer, a smart phone, a tablet computer, or a laptop computer, or a desktop computer, a projection computer, or other terminals, or a smart terminal with a data processing component. The embodiments of the present application do not limit this.

[0069] The server 120 can be implemented as a single server or as a server cluster composed of a group of servers. It can be a physical server or a cloud server. In a possible implementation, the server 120 is the background server of the application in the terminal 110.

[0070] In a possible implementation of this embodiment, the terminal 110 can obtain perception information and collect the electroencephalogram (EEG) of the driver during driving (such as the electroencephalogram signal containing brain waves). The terminal 110 makes a first driving mode decision based on the perception information and a second driving mode decision based on the EEG signal, obtaining a first decision result and a second decision result. After the terminal 110 obtains the first decision result and the second decision result, it sends the first decision result and the second decision result to the server 120 via a wired or wireless connection. The server 120 performs a fusion analysis on the first decision result and the second decision result, determines the control priority relationship between the first decision result and the second decision result, and then performs assisted driving control on the vehicle according to the fusion analysis result.

[0071] In another possible implementation of this embodiment, after the terminal 110 obtains the first decision result and the second decision result, it performs offline processing on the first decision result and the second decision result. The terminal 110 performs a fusion analysis on the first decision result and the second decision result, determines the control priority relationship between the first decision result and the second decision result, and then performs assisted driving control on the vehicle according to the fusion analysis result. The server 120 monitors the process of the terminal 110 processing data during assisted driving via a wired or wireless connection.

[0072] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0073] This embodiment will be described from the perspective of the assisted driving device. The assisted driving device can be specifically integrated in electronic devices such as servers or terminals. The terminal can include an electroencephalogram signal acquisition device, a PC, a tablet computer, a notebook computer, etc.

[0074] Figure 2 is a flowchart of an assisted driving method shown according to an exemplary embodiment. This method can be executed by a computer device, and the computer device can be the terminal 120 in the above Figure 1 shown embodiment. As Figure 2 shown, the process of the assisted driving method can include the following steps:

[0075] Step 201, obtain perception information, where the perception information includes vehicle environment information and vehicle state information.

[0076] In the embodiments of the present application, the so-called perception information refers to external data and vehicle own state data that can affect vehicle driving safety during the vehicle driving process, such as vehicle environment information and vehicle state information.

[0077] In a possible implementation manner, the perception information is the vehicle environment information and vehicle state information of the vehicle collected by a device with a perception module. The perception module includes a lidar, a camera, a navigation system, and a vehicle machine system, etc. The vehicle environment information may refer to all data related to vehicle driving safety in the environment where the vehicle is located collected by perception modules such as a lidar or a camera. For example, the vehicle environment information may be the driving states of all vehicles in the environment where the vehicle is located, or the driving environment in the environment where the vehicle is located (such as lane line data, obstacle data, real-time positioning data, and map data). Of course, the vehicle environment information may also be the traffic congestion state. The embodiments of the present application do not make special limitations on this. The vehicle state information may refer to all data related to the vehicle's own driving state during the vehicle's driving collected by perception modules such as a vehicle machine system. For example, the vehicle state information, actual vehicle speed data, sensing data on the steering wheel, brake pedal, accelerator pedal, and clutch. The embodiments of the present application do not make special limitations on this.

[0078] Step 202: Make a driving mode decision based on the perception information to obtain a first decision result.

[0079] In a possible implementation manner, the driving mode decision may be a vehicle driving control result made based on the vehicle environment information and vehicle state information. Preferably, the above-mentioned perception modules are pre-connected to the same device to facilitate information management of each perception module. The driving mode decision controls the vehicle's driving manner on the premise of ensuring driving safety.

[0080] Specifically, when the vehicle environment information is detected, obtain the picture resolution data, transmission period data, timestamp data, and color channel information corresponding to the vehicle environment information, associate the picture resolution data, transmission period data, timestamp data, and color channel information corresponding to the vehicle environment information with the vehicle driving information in the form of pictures. When the vehicle state information is detected, obtain the acquisition parameter data, acquisition period data, and timestamp data of the vehicle state information, associate the acquisition parameter data, acquisition period data, and timestamp data corresponding to the vehicle state information, and then associate the vehicle environment information and vehicle state information at the same moment to improve the calculation speed of the device, enhance the accuracy of the data, and further ensure the safety of the driving mode decision.

[0081] Step 203: Collect electroencephalogram signals of the driver to obtain electroencephalogram information.

[0082] In the embodiments of the present application, when brain nerve cells react, corresponding electric waves will be generated, and these electric waves are called brain waves. Collecting the brain waves and expressing them in the form of signals is the electroencephalogram signal.

[0083] In a possible implementation manner, the electroencephalogram signal may be the brain wave signal of a driver collected by a device with a brain-computer interface. The brain-computer interface has at least two electrodes. During the process of collecting signals from the driver through the brain-computer interface, the two electrodes are located in different regions of the driver's head so as to collect the brain wave signals generated in different regions of the driver.

[0084] Step 204: Classify the brain wave information to obtain a second decision result, and the second decision result is used to characterize the driving intention of the driver.

[0085] In a possible implementation manner, the second decision result may be a decision result based on identifying the driver state corresponding to the waveform of the brain wave information, so as to be used to characterize the driving intention of the driver. The driving intention includes the behavioral intentions of the driver such as accelerating, braking, steering, and shifting gears.

[0086] Specifically, when the brain wave information is detected, the trained convolutional neural network may be used to classify the brain wave information, and the trained convolutional neural network is used to predict the type of driving intention to which the brain wave signal belongs.

[0087] More specifically, a cosine similarity matrix is constructed according to the calculated cosine similarity, and the trained convolutional neural network is used to predict the type to which the cosine similarity matrix belongs to obtain the type of driving intention to which the brain wave signal belongs. For example, if the feature vectors of the brain wave information are all 128, then 128 cosine similarities can be calculated. At this time, a cosine similarity matrix can be constructed based on these 128 cosine similarities, and then the trained convolutional neural network is used to predict the type to which the cosine similarity matrix belongs to obtain the type of driving intention to which the brain wave signal belongs.

[0088] Step 205: Perform fusion analysis on the first decision result and the second decision result to obtain a fusion analysis result, and perform assisted driving control on the vehicle according to the fusion analysis result.

[0089] Figure 3 It is shown according to an exemplary embodiment Figure 2 The flowchart of step S202 in

[0090] Such as Figure 2 shown, Figure 2 The flowchart of step S202 in

[0091] Step S301: Perform feature extraction processing on the perception information to obtain an extraction result.

[0092] In a possible implementation manner, performing feature extraction processing on the perception information can be understood as calculating the corresponding index values of accuracy, latency, and feedback actions based on the original data of the collected perception information.

[0093] Specifically, perform feature extraction on the vehicle environment information to extract the index values corresponding to accuracy, latency, and feedback actions. The index value corresponding to accuracy can be the recognition accuracy rate and recognition recall rate of the vehicle environment information. To extract the recognition accuracy rate and recognition recall rate, it is necessary to determine the number of correct recognitions and the number of incorrect recognitions based on the recognition results. The recognition accuracy rate can be obtained by dividing the number of correct recognitions by the total number of recognitions. A similar calculation method can be used for the recognition recall rate. The index value corresponding to latency can be the recognition performance (recognition speed), and it is necessary to calculate the recognition performance (recognition speed) based on the total number of recognitions and the time taken for each or all recognitions to be completed. The index value corresponding to the feedback action can be the change amount of the brake / throttle pedal, the steering wheel rotation angle, and the steering wheel rotation direction. When there are index values corresponding to the feedback in the vehicle environment information, they can be directly selected from the target driving data.

[0094] Step S302: Determine the driving area of the vehicle based on the extraction result, and set the driving route of the vehicle according to the driving area.

[0095] In a possible implementation manner, the driving area can be understood as the area where the vehicle can be located on the basis of ensuring vehicle driving safety and not violating traffic rules. The driving route can be understood as the lane that indicates the direction in which the vehicle should drive when entering the intersection. In a traffic intersection with heavy traffic, lane lines are generally drawn, and the driving route can be confirmed through the lane lines. The purpose is to clarify the driving direction, keep to the lanes, and relieve traffic pressure. Lane lines include white dotted lines, white solid lines, guiding indication lines, deceleration reminder lines, and so on.

[0096] Determining the driving area of the vehicle based on the extraction results can be determined according to the distance between the vehicle and the vehicle in front and the distances between the vehicle and the obstacles on its left and right sides. By using radar ranging or camera shooting, the driving area of the vehicle is divided according to the distances between the vehicle and the obstacles in various directions determined by radar ranging, or the driving area of the vehicle is divided by identifying the distances between the vehicle and the obstacles in various directions in the image captured by the camera. Setting the driving route of the vehicle according to the driving area can be to continue moving forward in the original lane or change lanes according to the actual driving space in the driving area. For example, when the distance between the vehicle and the vehicle in front is less than the preset following preset value and the distance between the vehicle and the obstacle on its left side is greater than the preset adjacent preset value, the left lane is selected as the driving route and the vehicle changes lanes to the left lane.

[0097] Step S303, perform a status evaluation process on the vehicle according to the driving area and the driving route to obtain a status evaluation result, and select a driving mode according to the status evaluation result. The driving mode includes any one of constant speed driving, following driving, and emergency braking.

[0098] In a possible implementation, the status evaluation result can be understood as the result of classifying and selecting the driving status of the vehicle based on the current driving environment of the vehicle. For example, when the driving area reflects that the distance between the vehicle and the vehicle in front is greater than the preset following preset value, constant speed driving or following driving can be selected. When the driving area reflects that the distance between the vehicle and the vehicle in front is less than the preset following preset value, emergency braking can be selected.

[0099] Step S304, output a control command based on the selected driving mode as the first decision result.

[0100] In a possible implementation, the device with the status evaluation function interacts with the vehicle control system. The device sends the selected driving mode to the vehicle control system, and the vehicle control system converts the received driving mode into a control command for the vehicle to control the vehicle to drive according to the selected driving mode.

[0101] Figure 4 is shown according to an exemplary embodiment Figure 3 The flowchart of step S304 in

[0102] As Figure 4 shown, Figure 3 The flowchart of step S304 in

[0103] Step S401, when the driving mode is selected as following driving, divide the current following area according to the driving area to obtain a following driving area and a collision avoidance area.

[0104] In a possible implementation, dividing the current following driving area according to the driving area may be to preset the range of the collision avoidance area, including the distance in the driving direction and the distance in the steering direction. After determining the collision avoidance area, the remaining area of the driving area is divided into the following driving area.

[0105] For the following driving area, the following relationship is satisfied:

[0106] d stop <d actual ≤d stop +d follow ;

[0107] For the collision avoidance area, the following relationship is satisfied:

[0108] d actual <d stop ;

[0109] Wherein, d stop represents the distance of the collision avoidance area in the driving direction, d actual represents the actual distance between the vehicle and the vehicle in front, d follow represents the distance of the following driving area in the driving direction.

[0110] For example, in the following driving mode, the preset distance of the collision avoidance area in the driving direction is 5m. When there is a vehicle driving within 20m in front of the vehicle, 15m outside the distance of the collision avoidance area in the driving direction is set as the following driving area.

[0111] Step S402, determine the area where the vehicle is currently located.

[0112] Step S403, if the vehicle is in the following driving area, adjust the target speed of the vehicle during driving according to the actual distance between the vehicle and the vehicle in front, so that the target speed maintains a negative correlation with the actual distance between the vehicle and the vehicle in front.

[0113] In a possible implementation, adjusting the target speed of the vehicle during driving according to the actual distance between the vehicle and the vehicle in front, so that the target speed maintains a negative correlation with the actual distance between the vehicle and the vehicle in front, may be to periodically collect the actual speed of the vehicle and the actual distance between the vehicle and the vehicle in front. When the actual distance between the vehicle and the vehicle in front collected in the current collection period is less than the actual distance between the vehicle and the vehicle in front collected in the previous collection period, increase the target speed of the vehicle during driving, and use the increased target speed as the actual speed of the vehicle driving. On the contrary, when the actual distance between the vehicle and the vehicle in front collected in the current collection period is greater than the actual distance between the vehicle and the vehicle in front collected in the previous collection period, decrease the target speed of the vehicle during driving, and use the decreased target speed as the actual speed of the vehicle driving.

[0114] Exemplarily, when the actual distance between the vehicle and the vehicle in front collected in the current collection cycle is less than the actual distance between the vehicle and the vehicle in front collected in the previous collection cycle, the target speed satisfies:

[0115] v d = v d-1 + 1;

[0116] When the actual distance between the vehicle and the vehicle in front collected in the current collection cycle is greater than the actual distance between the vehicle and the vehicle in front collected in the previous collection cycle, the target speed satisfies:

[0117] v d = v d-1 - 1;

[0118] Wherein, v d represents the target speed of the previous collection cycle, v d-1 represents the target speed of the previous collection cycle, v d < v0, where v0 represents the following vehicle speed threshold.

[0119] Step S404, if the vehicle is in the collision avoidance area, adjust the target speed to zero.

[0120] In some embodiments, when the driving mode is uniform driving, the actual distance between the vehicle and the vehicle in front is within the preset range of uniform driving, and the vehicle is controlled to drive at a preset uniform driving speed.

[0121] Figure 5 is a schematic flow chart of Figure 2 step S204 shown according to an exemplary embodiment.

[0122] As Figure 5 shown, Figure 2 the process of step S204 in

[0123] Step S501, preprocess the electroencephalogram information to obtain the preprocessed electroencephalogram information.

[0124] In one possible implementation, preprocessing the electroencephalogram information may include signal processing processes such as denoising, shaping, filtering, and signal amplification processing.

[0125] Step S502, use the trained convolutional neural network to classify and test the preprocessed electroencephalogram information to obtain a classification result.

[0126] In one possible implementation, step S502 includes:

[0127] Perform labeling processing on the preprocessed electroencephalogram information to obtain labeled electroencephalogram information;

[0128] The labeled brainwave information is input into the trained convolutional neural network in the form of frames for classification testing. The output results of the trained convolutional neural network are rearranged according to the labels of the labeled brainwave information as the classification results.

[0129] The trained convolutional neural network consists of 5 layers, including: 1) an input layer, 2) a first convolutional layer, 3) a second convolutional layer, 4) a fully connected layer, and 5) an output layer. The input brainwave information is in the form of a 200*16 matrix, and the input of the fully connected layer is the output of the last convolutional layer:

[0130] y = sigmoid(WM + b);

[0131] Among them, y represents the output of the fully connected layer, W represents the weight matrix, b represents the bias, and M represents the output of the last convolutional layer.

[0132] After obtaining the labels of each brainwave information signal segment in the labeled brainwave information sample, the labeled brainwave information is classified by the trained convolutional neural network to determine the driving intention of the driver. A cosine similarity matrix is constructed based on the calculated cosine similarity. The type to which the cosine similarity matrix belongs is predicted by the convolutional neural network to obtain the driving intention type to which the brainwave signal sample belongs. The output results of the trained convolutional neural network are rearranged according to the labels of each previously obtained brainwave information signal segment, and then the classification results are obtained.

[0133] Step S503, based on the classification results, identify the driving intention of the driver to obtain an identification result, and convert the identification result into a control command as the second decision result.

[0134] In a possible implementation, the device with the brainwave information classification function interacts with the vehicle control system. The device sends the identification result to the vehicle control system, and the vehicle control system converts the received identification result into a control command for the vehicle and controls the vehicle to drive according to the control command.

[0135] Figure 6 It is shown according to an exemplary embodiment Figure 2 The flowchart of step S205 in

[0136] As Figure 6 shown, Figure 2 The flowchart of step S205 in

[0137] Step S601, identify the braking intention of the driver in the second decision result.

[0138] Step S602, if the second decision result includes a braking intention, then use the second decision result as the fusion analysis result.

[0139] In step S603, if the second decision result does not include a braking intention, the first decision result is used as the fusion analysis result.

[0140] In step S604, based on the fusion analysis result, a preset control output model is used to calculate a control output quantity, and the vehicle is assisted in driving control according to the control output quantity.

[0141] It can be understood that the fusion analysis result comprehensively analyzes the two decision-making methods of the first decision result and the second decision result. If the braking intention of the driver is recognized, emergency braking control is performed regardless of the control instruction corresponding to the second decision result. Otherwise, the control instruction corresponding to the second decision result is trusted.

[0142] In a possible implementation manner, step S604 includes:

[0143] Construct a control output model based on the target speed, and the control output model is:

[0144]

[0145] where U t represents the control output quantity, e(t) = v d -v a ,v d represents the target speed, v a represents the current speed, K P represents the proportional amplification coefficient, K I represents the integral time constant, K D represents the differential time constant;

[0146] Discretize the control output model to obtain a discrete control output model, and the discrete control output model is:

[0147]

[0148] where T is the instruction cycle of the controller, e j represents the speed deviation at the jth moment, e k represents the speed deviation at the kth moment, e k-1 represents the speed deviation at the (k - 1)th moment;

[0149] Use the discrete control output model to calculate the control output quantity, and perform assisted driving control on the vehicle through the calculated control output quantity.

[0150] As can be seen from the above, in the assisted driving method provided by the embodiments of the present application, on the one hand, perception information is obtained to determine the vehicle environment and vehicle state, and based on this, a decision result of the first type of assisted control is made. On the other hand, the electroencephalogram signals of the driver are collected to determine the driving intention of the driver, and based on this, a decision result of the second type of assisted control is made. Furthermore, the decision results of the two types of assisted control are combined to control the vehicle, so that the controlled vehicle makes a response under the combined influence of objective conditions and the subjective conditions of the driver, improving the safety of intelligent driving and enhancing the user experience.

[0151] To better implement the above method, an embodiment of the present invention further provides an assisted driving device, which can be specifically integrated in electronic devices such as servers or terminals.

[0152] Please refer to Figure 7 , an embodiment of the present application further provides an assisted driving device, which can implement the assisted driving method mentioned in the above embodiment. The device includes:

[0153] A perception module, configured to obtain perception information, where the perception information includes vehicle environment information and vehicle state information;

[0154] A first decision module, configured to make a driving mode decision according to the perception information to obtain a first decision result;

[0155] An electroencephalogram acquisition module, configured to collect electroencephalogram signals of the driver to obtain electroencephalogram information;

[0156] A second decision module, configured to perform classification processing on the electroencephalogram information to obtain a second decision result, where the second decision result is used to represent the driving intention of the driver;

[0157] An analysis module, configured to perform fusion analysis on the first decision result and the second decision result to obtain a fusion analysis result, and perform assisted driving control on the vehicle according to the fusion analysis result.

[0158] The specific implementation manner of this assisted driving device is basically the same as that of the specific embodiment of the above assisted driving method, and will not be elaborated here.

[0159] An embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above assisted driving method is implemented. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0160] Please refer to Figure 8 , Figure 8 illustrates the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0161] The processor 801 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0162] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 802 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 802 and are called by the processor 801 to execute the assisted driving method of the embodiments of the present application;

[0163] The input / output interface 803 is used to implement information input and output;

[0164] The communication interface 804 is used to implement communication interaction between this device and other devices, and can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0165] The bus 905 transmits information between the various components of the device (such as the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0166] Among them, the processor 801, the memory 802, the input / output interface 803, and the communication interface 804 are communicatively connected to each other inside the device through the bus 905.

[0167] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned assisted driving method is implemented.

[0168] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely provided relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The assisted driving method, device, electronic device, and storage medium provided by the embodiments of the present application, on the one hand, determine the vehicle environment and vehicle state by obtaining perception information, and make a decision result of the first type of assisted control based on this. On the other hand, determine the driving intention of the driver by collecting the electroencephalogram signal of the driver, and make a decision result of the second type of assisted control based on this. Furthermore, combine the decision results of the two types of assisted control to control the vehicle, so that the controlled vehicle makes a response under the combined influence of objective conditions and the driver's subjective conditions, improving the safety of intelligent driving and enhancing the user experience.

[0170] The embodiments described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0171] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine some steps, or different steps.

[0172] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations.

[0174] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0175] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist simultaneously. Here, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a, b, and c", where a, b, and c can be single or multiple.

[0176] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned unit division is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0177] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0179] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0180] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, which does not limit the scope of the rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall be within the scope of the rights of the embodiments of this application.

Claims

1. An assisted driving method, characterized in that, Including: Obtain perception information, where the perception information includes vehicle environment information and vehicle state information; Make a driving mode decision based on the perception information to obtain a first decision result; Collect electroencephalogram signals of the driver to obtain electroencephalogram information; Perform classification processing on the electroencephalogram information to obtain a second decision result, where the second decision result is used to represent the driving intention of the driver; Perform fusion analysis on the first decision result and the second decision result to obtain a fusion analysis result, and perform assisted driving control on the vehicle according to the fusion analysis result; The making a driving mode decision based on the perception information to obtain a first decision result includes: Perform feature extraction processing on the perception information to obtain an extraction result; Determine the driving area of the vehicle based on the extraction result, and set the driving route of the vehicle according to the driving area; Perform a state evaluation process on the vehicle according to the driving area and the driving route to obtain a state evaluation result, and select a driving mode according to the state evaluation result. The driving mode includes any one of constant speed driving, following vehicle driving, and emergency braking; Output a control command based on the selected driving mode as the first decision result; The performing classification processing on the electroencephalogram information to obtain a second decision result includes: Perform preprocessing on the electroencephalogram information to obtain preprocessed electroencephalogram information; Use a trained convolutional neural network to perform classification testing on the preprocessed electroencephalogram information to obtain a classification result; Identify the driving intention of the driver based on the classification result to obtain an identification result, and convert the identification result into a control command as the second decision result; The performing fusion analysis on the first decision result and the second decision result to obtain a fusion analysis result, and performing assisted driving control on the vehicle according to the fusion analysis result includes: Identify the braking intention of the driver in the second decision result; If the second decision result includes a braking intention, use the second decision result as the fusion analysis result; If the second decision result does not include a braking intention, use the first decision result as the fusion analysis result; Based on the fusion analysis result, use a preset control quantity output model to calculate a control output quantity, and perform assisted driving control on the vehicle according to the control output quantity.

2. The assisted driving method according to claim 1, wherein The outputting a control command based on the selected driving mode as the first decision result includes: When the driving mode is selected as following vehicle driving, divide the current following vehicle area according to the driving area to obtain a following vehicle driving area and a collision avoidance area; Judge the area where the vehicle is currently located; If the vehicle is in the following vehicle driving area, adjust the target speed when the vehicle is driving according to the actual distance between the vehicle and the vehicle in front, so that the target speed maintains a negatively correlated relationship with the actual distance between the vehicle and the vehicle in front; If the vehicle is in the collision avoidance area, adjust the target speed to zero.

3. The assisted driving method according to claim 1, wherein The using a trained convolutional neural network to perform classification testing on the preprocessed electroencephalogram information to obtain a classification result includes: Perform tagging processing on the preprocessed electroencephalogram information to obtain tagged electroencephalogram information; Input the tagged electroencephalogram information in the form of frames into the trained convolutional neural network for classification testing, and rearrange the output results of the trained convolutional neural network according to the tags of the tagged electroencephalogram information as the classification results.

4. The assisted driving method according to claim 1, wherein Based on the fusion analysis result, use a preset control output model to calculate the control output quantity, and perform assisted driving control on the vehicle according to the control output quantity, including: Construct a control output model based on the target speed, and the control output model is: ； Among them, U t represents the control output, e(t)=v d -v a , v d represents the target speed, v a represents the current speed, K P represents the proportional amplification coefficient, K I represents the integral time constant, K D represents the differential time constant; Perform discretization processing on the control output model to obtain a discrete control output model, and the discrete control output model is: ； where T is the instruction cycle of the controller, e j represents the speed deviation at time j, e k represents the speed deviation at time k, e k-1 represents the speed deviation at time k-1; Use the discrete control output model to calculate the control output quantity, and perform assisted driving control on the vehicle through the calculated control output quantity.

5. An assisted driving device, characterized in that, The device includes: A sensing module for acquiring sensing information, where the sensing information includes vehicle environment information and vehicle state information; A first decision-making module for making a driving mode decision according to the sensing information to obtain a first decision result; An electroencephalogram acquisition module for acquiring electroencephalogram signals of the driver to obtain electroencephalogram information; A second decision-making module for classifying the electroencephalogram information to obtain a second decision result, and the second decision result is used to represent the driving intention of the driver; An analysis module for fusing and analyzing the first decision result and the second decision result to obtain a fusion analysis result, and performing assisted driving control on the vehicle according to the fusion analysis result; The first decision-making module includes: A module for performing feature extraction processing on the sensing information to obtain an extraction result; A module for determining the driving area of the vehicle based on the extraction result and setting the driving route of the vehicle according to the driving area; A module for performing state evaluation processing on the vehicle according to the driving area and the driving route to obtain a state evaluation result, and selecting a driving mode according to the state evaluation result, where the driving mode includes any one of uniform speed driving, following vehicle driving, and emergency braking; A module for outputting a control instruction based on the selected driving mode as the first decision result; The second decision-making module includes: A module for preprocessing the electroencephalogram information to obtain preprocessed electroencephalogram information; A module for using a trained convolutional neural network to perform classification testing on the preprocessed electroencephalogram information to obtain a classification result; A module for identifying the driving intention of the driver based on the classification result to obtain an identification result, and converting the identification result into a control instruction as the second decision result; The analysis module includes: A module for identifying the braking intention of the driver in the second decision result; A module for using the second decision result as the fusion analysis result if the second decision result includes a braking intention; A module for using the first decision result as the fusion analysis result if the second decision result does not include a braking intention; A module for calculating a control output quantity by using a preset control quantity output model based on the fusion analysis result, and performing assisted driving control on the vehicle according to the control output quantity.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the assisted driving method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the assisted driving method according to any one of claims 1 to 4.

Images