Vehicle control method, vehicle, storage medium and product

Abstract

The invention discloses a vehicle control method, a vehicle, a storage medium and a product, and relates to the technical field of vehicles, the method comprises the following steps: obtaining feedback information made by a passenger in the vehicle based on a driving environment of the vehicle; and under the condition of determining that the vehicle is in the risk state according to the feedback information, determining to perform risk avoidance control on the vehicle based on the feedback information. Namely, the vehicle determines whether the vehicle is in the risk state according to the feedback information made by the vehicle passenger based on the driving environment, and performs corresponding risk avoidance control according to the feedback information when the vehicle is determined to be in the risk state. It can be understood that compared with a related scheme in which a vehicle-mounted radar, an ultrasonic sensor and other traditional acquisition devices are used for decision making, the feedback information made by a passenger according to the driving environment is used as the decision making basis of automatic driving, the dimension of automatic driving decision making reference is increased, the stability of automatic driving is improved, and the driving safety is improved. And the vehicle use safety is ensured.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle control method, a vehicle, a storage medium and a product. Background Art

[0002] As the level of autonomous driving advances from L2 to L4 and even L5, vehicles no longer rely solely on driver intervention, but need to have the ability to make highly autonomous decisions and respond to emergencies. At present, vehicle autonomous driving is mainly achieved through relevant hardware acquisition equipment and software control algorithms deployed on the vehicle. Therefore, in actual applications, once the autonomous driving vehicle encounters hardware damage, such as vehicle-mounted radar failure, ultrasonic sensor failure, etc., due to problems with data collection in this case, it will seriously affect the decision-making results of autonomous driving, thereby putting passengers in a high safety risk situation.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a vehicle control method, vehicle, storage medium and product, aiming to solve the technical problem that when an autonomous driving vehicle encounters hardware damage, it will put passengers in a situation with high safety risks.

[0005] To achieve the above object, the present application proposes a vehicle control method, which comprises the following steps:

[0006] Acquiring feedback information made by an occupant in a vehicle based on the driving environment of the vehicle, wherein the feedback information includes voice feature data of the occupant and / or facial feature data of the occupant;

[0007] In a case where it is determined according to the feedback information that the vehicle is in a risky state, risk avoidance control is performed on the vehicle based on the feedback information.

[0008] Optionally, the step of obtaining feedback information provided by a vehicle occupant based on the driving environment of the vehicle includes:

[0009] collecting facial image data of the occupant, and detecting the facial orientation of the occupant based on the facial image data;

[0010] When the face is facing the vehicle window, the voice feature data of the occupant is extracted from the in-vehicle audio data of the vehicle, and / or the mouth feature data of the occupant and the eye feature data of the occupant are extracted from the facial image data to obtain the facial feature data.

[0011] Optionally, after the step of obtaining feedback information from a vehicle occupant based on the driving environment of the vehicle, the method includes:

[0012] Determining whether the occupant has an emergency response based on the voice feature data and / or the mouth feature data and eye feature data in the facial feature data;

[0013] When it is determined that the occupant has an emergency reaction, it is determined that the vehicle is in a risky state.

[0014] Optionally, the step of determining whether the occupant has an emergency response based on the voice feature data and / or the mouth feature data and eye feature data in the facial feature data includes:

[0015] Comparing the voice feature data, and / or the mouth feature data and the eye feature data with preset emergency response index conditions, wherein the preset emergency response index conditions include a decibel index condition, an audio index condition and a timbre index condition corresponding to the voice feature data, an opening and closing index condition corresponding to the mouth feature data, and an eye movement index condition corresponding to the eye feature data;

[0016] When the preset emergency response index condition is met, it is determined that the occupant has an emergency response.

[0017] Optionally, the step of performing risk avoidance control on the vehicle based on the feedback information includes:

[0018] Determining a predicted risk source direction of the vehicle based on the feedback information;

[0019] When the predicted risk source direction is different from the detected risk source direction of the vehicle, risk avoidance control is performed on the vehicle based on the predicted risk source direction, wherein the risk avoidance control includes changing lanes to a lane away from the predicted risk source direction, and / or reducing the speed of the vehicle.

[0020] Optionally, the feedback information further includes the facial orientation of the occupant, and the step of determining the predicted risk source direction of the vehicle based on the feedback information includes:

[0021] When there are multiple occupants in the vehicle, determining an intersection point of the facial orientations of the occupants;

[0022] The predicted risk source direction is determined according to the direction of the intersection.

[0023] Optionally, after the step of determining the intersection of the facial orientations of the occupants, the method comprises:

[0024] In the case where there are multiple intersection points, determining the angle difference between the directions of the intersection points;

[0025] When the angle difference is greater than a preset angle threshold, the predicted risk source direction is determined according to the facial orientation of a target occupant located at a preset position among the occupants.

[0026] In addition, to achieve the above-mentioned purpose, the present application also proposes a vehicle control device, the vehicle control device comprising:

[0027] An acquisition module, used for acquiring feedback information made by an occupant in a vehicle based on the driving environment of the vehicle, wherein the feedback information includes voice feature data of the occupant and / or facial feature data of the occupant;

[0028] An avoidance module is used to perform risk avoidance control on the vehicle based on the feedback information when it is determined that the vehicle is in a risky state according to the feedback information.

[0029] In addition, to achieve the above objectives, the present application also proposes a vehicle, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the vehicle control method as described above.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the vehicle control method described above are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the vehicle control method described above are implemented.

[0032] One or more technical solutions proposed in this application have at least the following technical effects:

[0033] In the embodiment of the present application, feedback information made by the occupants in the vehicle based on the driving environment of the vehicle will be obtained, wherein the feedback information includes the voice feature data of the occupants and / or the facial feature data of the occupants; when the vehicle is determined to be in a risky state according to the feedback information, risk avoidance control is performed on the vehicle based on the feedback information. That is, the vehicle will determine whether the vehicle is in a risky state according to the feedback information made by the vehicle occupants based on the driving environment, including the voice feature data of the occupants and the facial feature data of the occupants, and make corresponding risk avoidance control according to the feedback information when it is determined to be in a risky state. It can be understood that compared with the decision-making of traditional acquisition devices such as vehicle-mounted radar and ultrasonic sensors in the relevant scheme, the embodiment of the present application also uses the feedback information made by the occupants based on the driving environment as the basis for decision-making, increases the dimension of vehicle control decision reference, and improves the stability of autonomous driving. Therefore, even if the acquisition equipment related to autonomous driving fails or fails, the present application can still make risk avoidance control according to the feedback information of the occupants, thereby improving the safety of the vehicle as much as possible and protecting the personal and property safety of the owner. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 This is a flow chart of the first embodiment of the vehicle control method of the present application;

[0037] Figure 2 This is a flow chart of the second embodiment of the vehicle control method of the present application;

[0038] Figure 3 This is a flow chart of the third embodiment of the vehicle control method of the present application;

[0039] Figure 4 This is a schematic diagram of the overall process framework in the vehicle control method of this application;

[0040] Figure 5 This is a schematic diagram of the structure corresponding to the vehicle control device of this application;

[0041] Figure 6Schematic diagram of the device structure of the hardware operating environment involved in the vehicle control method in the embodiment of the present application.

[0042] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0044] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0045] As the level of autonomous driving advances from L2 to L4 and even L5, vehicles no longer rely solely on driver intervention, but need to have the ability to make highly autonomous decisions and respond to emergencies. At present, vehicle autonomous driving is mainly achieved through relevant hardware acquisition equipment and software control algorithms deployed on the vehicle. Therefore, in actual applications, once the autonomous driving vehicle encounters hardware damage, such as sensor failure, due to problems with data collection in this case, it will seriously affect the decision-making results of autonomous driving, which will put passengers in a high safety risk situation.

[0046] The main solution of the embodiment of the present application is: obtaining feedback information made by the occupants in the vehicle based on the driving environment of the vehicle, wherein the feedback information includes the voice feature data of the occupants and / or the facial feature data of the occupants; and when it is determined that the vehicle is in a risky state according to the feedback information, performing risk avoidance control on the vehicle based on the feedback information.

[0047] That is, the vehicle will determine whether the vehicle is in a risky state based on the feedback information made by the vehicle occupants based on the driving environment, including the voice feature data of the occupants and the facial feature data of the occupants, and make corresponding risk avoidance control based on the feedback information when it is determined to be in a risky state. It can be understood that compared with the decision-making based on traditional acquisition equipment such as vehicle-mounted radar and ultrasonic sensors in the related schemes, the embodiment of the present application also uses the feedback information made by the occupants based on the driving environment as the basis for decision-making, which increases the dimension of reference for vehicle control decisions, that is, improves the stability of autonomous driving. Therefore, even if the acquisition equipment related to autonomous driving fails or fails, the present application can still make risk avoidance control based on the feedback information of the occupants, thereby improving the safety of the vehicle as much as possible and protecting the personal and property safety of the owner.

[0048] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a cloud platform, vehicle, computer, mobile phone, etc., or a vehicle that can realize the above functions.

[0049] Based on this, the embodiment of the present application provides a vehicle control method, referring to Figure 1 , which is a flow chart of the first embodiment of the vehicle control method of the present application.

[0050] In this embodiment, the vehicle control method includes steps S10 to S20:

[0051] Step S10, obtaining feedback information made by an occupant in the vehicle based on the driving environment of the vehicle, wherein the feedback information includes voice feature data of the occupant and / or facial feature data of the occupant;

[0052] In this embodiment, the execution subject of the above vehicle control method can be a vehicle. And the vehicle control method can be a part of the automatic driving control method. It can be understood that the current vehicle automatic driving is mainly realized by the relevant hardware acquisition equipment and software control algorithm deployed on the vehicle. If the hardware equipment fails, it may cause problems with the data input to the automatic driving control algorithm, thereby causing errors or failures in the decision-making of the automatic driving. Therefore, at this time, the vehicle and passengers will be placed in a high-risk non-situation. Therefore, in response to this situation, this embodiment will perform risk avoidance control based on the feedback information made by the occupants of the vehicle's driving environment, so as to improve the safety of the vehicle and protect the personal and property safety of the owner as much as possible in the case of automatic driving related hardware failure or failure (at this time, the execution subject vehicle of the automatic driving may not know that there is a problem with the automatic driving, so it will still be put into operation in the automatic driving state).

[0053] Exemplarily, the vehicle will collect feedback information made by the occupants of the vehicle according to the driving environment of the vehicle. For example, the feedback information may include voice feature data in the vehicle, and may also include facial feature data of the occupants inside the vehicle. Correspondingly, a microphone may be provided on the vehicle for collecting voice feature data of the occupants, and a camera may also be provided on the vehicle, and the camera is used to collect facial images of the occupants to obtain facial feature data, wherein the facial images of the occupants at least include facial images of the driver. In addition, for determining whether the voice feature data and facial feature data are made by the occupants based on the driving environment, it can be determined based on the facial features (i.e., facial images) of the occupants. For example, when the occupant's face is facing the window (i.e., outside the vehicle), it can be determined that the voice feature data and facial feature data collected at this time (and the preset time thereafter) are feedback information made according to the driving environment. It should be understood that the feedback information may include only the voice feature data of the occupants, only the facial feature data of the occupants, or both the voice feature data of the occupants and the facial feature data of the occupants. It can be specifically set by technical personnel according to actual needs, and preferably includes both voice feature data and facial feature data.

[0054] Step S20: When it is determined according to the feedback information that the vehicle is in a risky state, risk avoidance control is performed on the vehicle based on the feedback information.

[0055] It should be noted that in this embodiment, it is also determined whether the vehicle is in a risk state based on the feedback information. For example, the in-car audio data in the feedback information can be used to determine whether the occupant has made behaviors such as shouting, screaming or exclamation, and the facial image in the feedback information can also be used to determine whether the occupant has emergency expressions such as tension and panic. If the occupant has any of the above behaviors or expressions, it can be determined that the vehicle is in a risk state. Accordingly, the vehicle is subjected to risk avoidance control based on the feedback information. It can be understood that the main purpose of risk avoidance control of the vehicle is to keep the vehicle away from the risk source. If the feedback information includes in-car audio data, and it is determined through the in-car audio data that the occupant has screaming behavior, the corresponding risk avoidance control can be to control the vehicle to slow down or brake; if the feedback information includes the facial feature data of the occupant, the direction of the risk source can be predicted by the facial orientation of the occupant in the process of determining that the vehicle is in a risk state. Accordingly, the risk avoidance control performed can be to change lanes according to the direction of the risk source, so as to stay away from the risk source as much as possible and ensure the safety of the vehicle. It is understandable that, in the case where the feedback information includes in-car audio data and facial feature data, the technical personnel can refer to the above examples and actual needs to combine and set them, so they will not be elaborated here.

[0056] In this embodiment, feedback information made by the occupants in the vehicle based on the driving environment of the vehicle will be obtained, wherein the feedback information includes the voice feature data of the occupants and / or the facial feature data of the occupants; when the vehicle is determined to be in a risky state according to the feedback information, risk avoidance control is performed on the vehicle based on the feedback information. That is, the vehicle will determine whether the vehicle is in a risky state according to the feedback information made by the vehicle occupants based on the driving environment, including the voice feature data of the occupants and / or the facial feature data of the occupants, and make corresponding risk avoidance control according to the feedback information when it is determined to be in a risky state. It can be understood that compared with the decision-making of traditional acquisition devices such as vehicle-mounted radar and ultrasonic sensors in the relevant scheme, the embodiment of the present application also uses the feedback information made by the occupants based on the driving environment as the basis for decision-making, increases the dimension of vehicle control decision reference, and improves the stability of autonomous driving. Therefore, even if the acquisition equipment related to autonomous driving fails or fails, the present application can still make risk avoidance control according to the feedback information of the occupants, thereby improving the safety of the vehicle as much as possible and protecting the personal and property safety of the owner.

[0057] In a feasible implementation manner, the facial feature data includes mouth feature data and eye feature data, and the step of obtaining feedback information made by the occupant in the vehicle based on the driving environment of the vehicle includes steps S11 to S13:

[0058] Step S11, collecting facial image data of the occupant, and detecting the facial orientation of the occupant based on the facial image data;

[0059] Step S12, when the face is facing the direction of the vehicle window, the voice feature data of the occupant is extracted from the in-vehicle audio data of the vehicle, and / or the mouth feature data of the occupant and the eye feature data of the occupant are extracted from the facial image data to obtain the facial feature data.

[0060] For example, a microphone and a camera may be configured in the vehicle, the microphone is used to collect the in-vehicle audio data inside the vehicle, and the camera is used to collect the facial image data of the occupants. In practical applications, cameras may be set up in both the front and rear rows of the vehicle to collect the facial image data of the front and rear row occupants. In order to save costs, and considering that the driver pays more attention to the driving environment among the occupants, a camera for collecting the driver's facial image data may be set up first.

[0061] First, the facial orientation of the occupant can be detected based on the facial image data of the occupant. It is worth noting that the facial orientation can be detected through a pre-trained AI model. The AI ​​model can be an LVQ (Learning Vector Quantization) neural network model, and the features of the facial image data are input into the neural network model, and the facial orientation of the occupant in the image is identified through the neural network model. In the stage of model training, facial images of different orientations can be collected first, and facial feature vectors, especially feature vectors of eye positions, can be extracted. Then the image is preprocessed, including cropping, conversion to grayscale images, and edge detection using the Sobel edge operator to extract eye position information. The extracted feature vectors are used as the input of the LVQ neural network, and different orientations (such as left, left front, front, right front and right) are represented by different numbers as the output of the network, and then the parameters in the model are updated based on the difference between the output of the network and the image label. Referring to the above method, after the model is trained using the images in the training set, a model with prediction function can be obtained, and the orientation judgment and recognition of any given facial image can be performed.

[0062] In an optional embodiment, since the installation position and shooting angle of the camera that collects the facial image of the occupant are usually fixed in actual application, the installation position and shooting angle of the camera can be used as prior information, and then the facial orientation of the occupant can be calculated by combining the prior information and the facial features in the facial image. Take the occupant as an example, the camera is usually installed in front of the driver. Therefore, the more facial features are detected, the higher the driver's facial orientation is, and vice versa. As for the left and right directions, it can be judged according to the distribution position of the facial features in the facial image. For example, since the camera is installed in front of the driver, if the driver faces straight ahead, his facial features are close to symmetrical distribution in the facial image. Therefore, in actual detection, the orientation angle of the driver's face in the left and right directions is calculated according to the actual distribution of the facial features in the facial image and the deviation angle from the symmetrical distribution. In addition, considering that AI model recognition has a high demand for computing power, the real-time performance will be lower in the case of limited computing power equipment. Therefore, when the vehicle's computing power is limited, the above-mentioned method of combining prior information and facial features in the facial image to calculate the facial orientation of the occupant can be selected. When the computing power is sufficient, the above-mentioned method of identifying the facial orientation through the AI ​​model can be selected.

[0063] When determining whether the facial orientation is the direction of the car window, the main thing is to judge the occupant's facial orientation in the up and down direction. If the facial orientation is within a preset angle range in the up and down direction, it can be determined that the occupant's facial orientation is the direction of the car window. Correspondingly, it can be considered that the occupant's reaction at this time is made according to the driving environment. Therefore, when it is determined that the facial orientation is the direction of the car window, the voice feature data can be extracted from the in-car audio data, and the mouth feature data and eye feature data can be extracted from the facial image data, and then the voice feature data, and / or, the mouth feature data and the eye feature data are used as feedback information. Among them, the extraction of voice feature data is mainly to remove the environmental noise in the in-car audio data. It can be understood that there is an essential difference between environmental noise, normal passenger communication sound and emergency call sound, which can be judged by decibel size and timbre. The decibel, timbre, and audio regularity (peak value and change curve) of multiple emergency call sounds, normal communication sounds of personnel and environmental noise can be stored in the memory first. And the noise and human voice parameter numerical range are fitted as the extraction standard. In addition, the mouth feature data and eye feature data can be extracted by face recognition technology, and since the current face recognition technology is relatively mature, it will not be described here. It can be understood that the feedback information can include only voice feature data, only mouth feature data and eye feature data, or all voice feature data, mouth feature data and eye feature data.

[0064] Reference Figure 2 , is a flow chart of a second embodiment proposed based on the first embodiment of the vehicle control method of the present application. For the same or similar contents in this embodiment as those in the above embodiment, reference can be made to the above introduction and no further description will be given later. After the step of obtaining feedback information made by the occupants in the vehicle based on the driving environment of the vehicle, the method includes steps A10 to A20:

[0065] Step A10, determining whether the occupant has an emergency response based on the voice feature data and / or the mouth feature data and eye feature data in the facial feature data;

[0066] Step A20, when it is determined that the occupant has an emergency reaction, it is determined that the vehicle is in a risky state.

[0067] For example, in this embodiment, the vehicle will use the voice feature data, mouth feature data, and eye feature data in the feedback information to determine whether the occupant has an emergency reaction. Emergency reaction refers to the subconscious reaction of the occupant in an emergency, such as screaming, calling, opening eyes, opening mouth, and frowning. If it is determined that the occupant has the above reactions based on the voice feature data, mouth feature data, and eye feature data, it can be considered that the occupant has an emergency reaction, and accordingly, it is determined that the vehicle is in a risky state.

[0068] In a feasible implementation manner, the step of determining whether the occupant has an emergency response based on the voice feature data and / or the mouth feature data and eye feature data in the facial feature data includes steps A11 to A12:

[0069] Step A11, comparing the voice feature data, and / or the mouth feature data and the eye feature data with preset emergency response index conditions, wherein the preset emergency response index conditions include a decibel index condition, an audio index condition and a timbre index condition corresponding to the voice feature data, an opening and closing index condition corresponding to the mouth feature data, and an eye movement index condition corresponding to the eye feature data;

[0070] Step A12: determining that the occupant has an emergency response when the preset emergency response index condition is met.

[0071] Exemplarily, by comparing the voice feature data, and / or the mouth feature data and the eye feature data with the preset emergency response index conditions, it can be determined whether the occupant has an emergency response. Among them, the preset emergency response index conditions may include decibel index conditions, audio index conditions and timbre index conditions corresponding to the voice feature data, opening and closing index conditions corresponding to the mouth feature data, and eye movement index conditions corresponding to the eye feature data. The decibel index conditions, audio index conditions, timbre index conditions and opening and closing index conditions may be preset thresholds or ranges. Similarly, the normal communication sound of the occupant is essentially different from the emergency call sound, which can be judged by decibel size and timbre. First, the decibel, timbre, and audio rules (peak value and change curve) of multiple emergency calls, normal communication sounds of personnel, and environmental noise are stored in the memory. The above-mentioned stored data comes from recording 10 groups of data of different groups of people and different in-vehicle occasions through an audio collector, and fitting the parameter value interval of the emergency call sound and the normal communication sound of personnel (that is, the above-mentioned decibel index conditions, audio index conditions, and timbre index conditions) as the judgment standard. To avoid overlap of intervals, it is recommended that the sound sampling frequency be 2000HZ. On the other hand, when the driver makes an emergency call, his face often shows an open mouth. Therefore, the degree of opening and closing of the mouth when different groups of people call can also be counted to obtain the opening and closing index condition. In addition, the eye movement index condition can include frowning and opening eyes. It should be noted that in a feasible way, if any one of the relevant feature data of the occupant (voice feature data, mouth feature data or eye feature data) meets the preset emergency response index condition, it can be determined that the occupant has an emergency response. In addition, in practical applications, in order to ensure the accuracy of detection, the comprehensive relevant feature data can be judged, such as comparing the voice feature data with the decibel index condition, the audio index condition and the timbre index condition to obtain the score under this type of index condition (for example, the closer to the decibel index condition, the audio index condition and the timbre index condition, the higher the score). Similarly, referring to the above process, the mouth feature data is compared with the opening and closing index condition, and the eye feature data is compared with the eye movement index condition, and the scores of each index condition are obtained respectively, and then the scores of each index condition are accumulated to obtain the total score. If the total score is greater than the preset score, it is considered that the preset emergency response index condition is met, and accordingly, it can be determined that the occupant has an emergency response. Otherwise, it is considered that the preset emergency response index condition is not met, and it can be determined that the occupant does not have an emergency response, and no risk avoidance control is performed.

[0072] Reference Figure 3, is a flow chart of a third embodiment proposed based on the first embodiment and the second embodiment of the vehicle control method of the present application. The same or similar contents in this embodiment as those in the above embodiments can be referred to the above introduction and will not be described in detail later. The step of performing risk avoidance control on the vehicle based on the feedback information includes steps S21 to S22:

[0073] Step S21, determining the predicted risk source direction of the vehicle based on the feedback information;

[0074] Step S22, when the predicted risk source direction is different from the detected risk source direction of the vehicle, risk avoidance control is performed on the vehicle based on the predicted risk source direction, wherein the risk avoidance control includes changing lanes to a lane away from the predicted risk source direction, and / or reducing the speed of the vehicle.

[0075] Exemplarily, in this embodiment, the predicted risk source direction of the vehicle can be determined according to the feedback information. For example, when the vehicle is determined to be in a risk state according to the feedback information, the predicted risk source direction is determined according to the occupant's facial orientation in the feedback information or corresponding to the feedback information, such as the occupant's facial orientation can be directly used as the predicted risk source direction. After obtaining the predicted risk source direction, the predicted risk source direction can be compared with the detected risk source direction of the vehicle, wherein the detected risk source direction is the direction in which obstacles (such as other vehicles) are detected by the vehicle's sensors and radars. It can be understood that in this embodiment, the risk avoidance control of the vehicle based on the feedback information is mainly to ensure that obstacles can still be avoided to reduce vehicle safety risks when the hardware related to automatic driving in the vehicle fails or fails. Therefore, in order to avoid repeated control, the risk avoidance control of the vehicle can be performed based on the predicted risk source direction only when the predicted risk source direction is different from the detected risk source direction of the vehicle. Accordingly, the risk avoidance control includes changing lanes to a lane away from the predicted risk source direction, and / or reducing the speed of the vehicle. For example, assuming that the vehicle is traveling on a three-lane one-way road, and is divided into three lanes: left, middle and right. If the vehicle is in the right lane and the predicted risk source direction is forward, the vehicle can reduce the speed of the vehicle and try to change lanes from the right lane to the middle lane and the left lane in sequence; if the vehicle is in the middle lane and the predicted risk source direction is forward, the vehicle tries to change lanes from the middle lane to the left lane or the right lane; if the vehicle is in the left lane and the predicted risk source direction is right front, the vehicle can slow down and keep driving in the current lane, or perform emergency braking in the current lane. In addition, technicians can also set other risk avoidance strategies with reference to the above examples. In addition, it can be understood that risk avoidance control of the vehicle based on feedback information indicates that there may be abnormalities in the vehicle's automatic driving. Therefore, after risk avoidance control of the vehicle, the driver can also be prompted to take over the control of the vehicle, thereby reducing the safety risks brought by automatic driving.

[0076] In a feasible implementation manner, the feedback information further includes the facial orientation of the occupant, and the step of determining the predicted risk source direction of the vehicle based on the feedback information includes steps S211 to S212:

[0077] Step S211, when there are multiple passengers in the vehicle, determining the intersection of the facial orientations of the passengers;

[0078] Step S212, determining the predicted risk source direction according to the direction of the intersection.

[0079] For example, in some cases, there may be multiple passengers in the vehicle. In order to ensure the accuracy of the predicted risk source direction, in this embodiment, the predicted risk source direction will be determined in combination with the facial orientation of the passengers. For example, the focus of the facial orientation of the passengers can be determined first. Taking two passengers as an example, the facial orientations of the two passengers can be determined respectively, and then the intersection of the facial orientations of the two passengers can be determined. If there is an intersection, the direction of the predicted risk source can be directly determined by the direction of the intersection. If there is no intersection, the driver's facial orientation is used as the direction of the predicted risk source.

[0080] In a feasible implementation manner, after the step of determining the intersection of the facial orientations of the occupants, the method includes steps S213 to S214:

[0081] Step S213, when there are multiple intersections, determine the angle difference between the directions of the intersections;

[0082] Step S214, when the angle difference is greater than a preset angle threshold, the predicted risk source direction is determined according to the facial orientation of the target occupant located at a preset position among the occupants.

[0083] For example, if there are multiple intersections, the angle difference between the directions of the intersections can be determined. If the angle difference is large, it means that the facial orientations of the occupants are basically not uniform, that is, at this time, referring to the facial orientations of the occupants, there may be a large error in determining the predicted risk source direction. Therefore, in order to ensure the accuracy of the predicted risk source direction, the facial orientation of the target occupant at the preset position (that is, the driver's position) is directly used as the predicted risk source direction.

[0084] Reference Figure 4 , is a schematic diagram of the overall process framework of the vehicle control method in this application, Figure 4 The following will take the case where the passenger is the driver. Figure 4 As shown, the vehicle can monitor the driver's behavior in real time to determine whether the driver has made an emergency call, for example, through the voice data sent by the passengers inside the vehicle; the vehicle will also identify the driver's facial features in real time to determine whether there are any abnormalities in the driver's mouth and eyes. If there is an emergency call, or the facial features are abnormal, it can be considered that the driver has an emergency response, and accordingly, the automatic driving part of the vehicle may be abnormal, so the vehicle's ECU (Electronic Control Unit) can perform corresponding risk avoidance control behaviors and turn on the automatic emergency braking function, lane keeping function, and collision warning function.

[0085] The present application also provides a vehicle control device, referring to Figure 5 , the vehicle control device comprises:

[0086] An acquisition module 10, used for acquiring feedback information made by an occupant in a vehicle based on the driving environment of the vehicle, wherein the feedback information includes voice feature data of the occupant and / or facial feature data of the occupant;

[0087] The avoidance module 20 is configured to perform risk avoidance control on the vehicle based on the feedback information when it is determined that the vehicle is in a risky state according to the feedback information.

[0088] Optionally, the facial feature data includes mouth feature data and eye feature data, and the acquisition module 10 is further used for:

[0089] collecting facial image data of the occupant, and detecting the facial orientation of the occupant based on the facial image data;

[0090] When the face is facing the vehicle window, the voice feature data of the occupant is extracted from the in-vehicle audio data of the vehicle, and / or the mouth feature data of the occupant and the eye feature data of the occupant are extracted from the facial image data to obtain the facial feature data.

[0091] Optionally, the vehicle control device further includes a determination module 30, wherein the determination module is configured to:

[0092] Determining whether the occupant has an emergency response based on the voice feature data and / or the mouth feature data and eye feature data in the facial feature data;

[0093] When it is determined that the occupant has an emergency reaction, it is determined that the vehicle is in a risky state.

[0094] Optionally, the determining module 30 is further configured to:

[0095] Comparing the voice feature data, and / or the mouth feature data and the eye feature data with preset emergency response index conditions, wherein the preset emergency response index conditions include a decibel index condition, an audio index condition and a timbre index condition corresponding to the voice feature data, an opening and closing index condition corresponding to the mouth feature data, and an eye movement index condition corresponding to the eye feature data;

[0096] When the preset emergency response index condition is met, it is determined that the occupant has an emergency response.

[0097] Optionally, the avoidance module 20 is used to:

[0098] Determining a predicted risk source direction of the vehicle based on the feedback information;

[0099] When the predicted risk source direction is different from the detected risk source direction of the vehicle, risk avoidance control is performed on the vehicle based on the predicted risk source direction, wherein the risk avoidance control includes changing lanes to a lane away from the predicted risk source direction, and / or reducing the speed of the vehicle.

[0100] Optionally, the feedback information further includes the facial orientation of the occupant, and the avoidance module 20 is configured to:

[0101] When there are multiple occupants in the vehicle, determining an intersection point of the facial orientations of the occupants;

[0102] The predicted risk source direction is determined according to the direction of the intersection.

[0103] Optionally, the avoidance module 20 is used to:

[0104] In the case where there are multiple intersection points, determining the angle difference between the directions of the intersection points;

[0105] When the angle difference is greater than a preset angle threshold, the predicted risk source direction is determined according to the facial orientation of a target occupant located at a preset position among the occupants.

[0106] The vehicle control device provided by the present application adopts the vehicle control method in the above embodiment, which can solve the technical problem that when the autonomous driving vehicle encounters hardware damage, it will put passengers in a situation with high safety risks. Compared with the prior art, the beneficial effects of the vehicle control device provided by the present application are the same as the beneficial effects of the vehicle control method provided by the above embodiment, and the other technical features in the vehicle control device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0107] The present application provides a vehicle, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vehicle control method in the above-mentioned embodiment one.

[0108] Reference below Figure 6, which shows a schematic diagram of the structure of a vehicle suitable for implementing the embodiment of the present application. The vehicle in the embodiment of the present application may include but is not limited to mobile terminals such as computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as computers. Figure 6 The vehicle shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 6 As shown, the vehicle may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for vehicle operation are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the vehicle to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a vehicle with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0110] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0111] The vehicle provided by this application adopts the vehicle control method in the above embodiment, which can solve the technical problem that when the autonomous driving vehicle encounters hardware damage, it will put passengers in a situation with high safety risks. Compared with the prior art, the beneficial effects of the vehicle provided by this application are the same as the beneficial effects of the vehicle control method provided by the above embodiment, and the other technical features in the vehicle are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0112] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0113] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0114] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the vehicle control method in the above-mentioned embodiment.

[0115] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0116] The computer-readable storage medium may be included in the vehicle, or may exist independently without being installed in the vehicle.

[0117] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a vehicle, the vehicle:

[0118] Acquiring feedback information made by an occupant in a vehicle based on the driving environment of the vehicle, wherein the feedback information includes voice feature data of the occupant and / or facial feature data of the occupant;

[0119] In a case where it is determined according to the feedback information that the vehicle is in a risky state, risk avoidance control is performed on the vehicle based on the feedback information.

[0120] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0123] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned vehicle control method, and can solve the technical problem that when the autonomous driving vehicle encounters hardware damage, it will put passengers in a situation of high safety risk. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the vehicle control method provided by the above-mentioned embodiment, and will not be repeated here.

[0124] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the vehicle control method as described above are implemented.

[0125] The computer program product provided by the present application can solve the technical problem of vehicle control. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the vehicle control method provided by the above embodiment, which will not be repeated here.

[0126] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A vehicle control method, characterized in that: The vehicle control method comprises the following steps: Acquiring feedback information made by an occupant in a vehicle based on the driving environment of the vehicle, wherein the feedback information includes voice feature data of the occupant and / or facial feature data of the occupant; In a case where it is determined according to the feedback information that the vehicle is in a risky state, risk avoidance control is performed on the vehicle based on the feedback information.

2. The vehicle control method according to claim 1, characterized in that: The step of obtaining feedback information provided by a vehicle occupant based on the driving environment of the vehicle includes: collecting facial image data of the occupant, and detecting the facial orientation of the occupant based on the facial image data; When the face is facing the vehicle window, the voice feature data of the occupant is extracted from the in-vehicle audio data of the vehicle, and / or the mouth feature data of the occupant and the eye feature data of the occupant are extracted from the facial image data to obtain the facial feature data.

3. The vehicle control method according to claim 1, characterized in that: After the step of obtaining feedback information from a vehicle occupant based on the driving environment of the vehicle, the method includes: Determining whether the occupant has an emergency response based on the voice feature data and / or the mouth feature data and eye feature data in the facial feature data; When it is determined that the occupant has an emergency reaction, it is determined that the vehicle is in a risky state.

4. The vehicle control method according to claim 3, characterized in that: The step of determining whether the occupant has an emergency response based on the voice feature data and / or the mouth feature data and eye feature data in the facial feature data comprises: Comparing the voice feature data, and / or the mouth feature data and the eye feature data with preset emergency response index conditions, wherein the preset emergency response index conditions include a decibel index condition, an audio index condition and a timbre index condition corresponding to the voice feature data, an opening and closing index condition corresponding to the mouth feature data, and an eye movement index condition corresponding to the eye feature data; When the preset emergency response index condition is met, it is determined that the occupant has an emergency response.

5. The vehicle control method according to claim 1, characterized in that: The step of performing risk avoidance control on the vehicle based on the feedback information comprises: Determining a predicted risk source direction of the vehicle based on the feedback information; When the predicted risk source direction is different from the detected risk source direction of the vehicle, risk avoidance control is performed on the vehicle based on the predicted risk source direction, wherein the risk avoidance control includes changing lanes to a lane away from the predicted risk source direction, and / or reducing the speed of the vehicle.

6. The vehicle control method according to claim 5, characterized in that: The feedback information further includes the facial orientation of the occupant, and the step of determining the predicted risk source direction of the vehicle based on the feedback information includes: When there are multiple occupants in the vehicle, determining an intersection point of the facial orientations of the occupants; The predicted risk source direction is determined according to the direction of the intersection.

7. The vehicle control method according to claim 6, characterized in that: After the step of determining the intersection of the facial orientations of the occupants, the method includes: In the case where there are multiple intersection points, determining the angle difference between the directions of the intersection points; When the angle difference is greater than a preset angle threshold, the predicted risk source direction is determined according to the facial orientation of a target occupant located at a preset position among the occupants.

8. A vehicle, characterized in that: The vehicle comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the vehicle control method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the vehicle control method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the vehicle control method according to any one of claims 1 to 7 are implemented.

Images