In-vehicle information interaction method based on voice recognition, in-vehicle infotainment system and vehicle

Abstract

The invention discloses an in-vehicle information interaction method based on voice recognition, an in-vehicle infotainment system and a vehicle, and relates to the technical field of information interaction, and the method comprises the steps: collecting sound training data, and training a tone recognition model for recognizing the tone category based on the sound training data; obtaining in-vehicle sound data, framing the sound data to obtain m groups of framing audios, extracting a voice instruction in the sound data, inputting the framing audios corresponding to the voice instruction into a tone recognition model, and outputting a tone category; and acquiring environment data in real time, and generating an intelligent interaction instruction based on the voice instruction, the tone category and the environment data. The reasonability of the voice instruction is evaluated in combination with the environment data, whether the initial instruction is generated or not is judged, calculation is performed based on the initial instruction to generate the intelligent interaction instruction conforming to the current environment data to control the vehicle, and the safety and intelligence of information interaction are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of information interaction, and in particular relates to a method for in-vehicle information interaction based on voice recognition, a vehicle system and a vehicle. Background Art

[0002] Currently, in-vehicle information interaction is mainly passive voice interaction, that is, the people in the car send voice commands to wake up the car computer for execution.

[0003] For example, the Chinese patent with publication number CN117275488A discloses a method for in-vehicle information interaction, a vehicle system and a vehicle. The method includes: obtaining sound information in the vehicle; converting the sound information into text information; extracting keywords and abstracts from the text information; inputting the extracted keywords and abstracts into a generative AI module; and the generative AI module fusing the keywords and abstracts to generate derivative content of images or videos.

[0004] This patent extracts keywords from the sound information in the car and combines them with AI to generate related images or videos for playback, thereby enhancing the fun of the car. However, when driving, the car computer playing related images or videos will attract the driver, causing the driver to be distracted and affecting driving safety. In addition, the current in-car information interaction is all passive and cannot be combined with the current environmental characteristics for intelligent interaction. For example, when a vehicle is driving at high speed, after receiving a window opening command, the window will be opened directly. However, suddenly opening the window of a high-speed vehicle will affect the vehicle's airflow, and items in the car will suddenly be blown away, affecting driving safety.

[0005] For another example, a Chinese patent with publication number CN108831460A discloses a voice interaction control system based on fatigue monitoring, including: a fatigue monitoring module, used to monitor the user's bad driving behavior during vehicle driving; a fatigue level judgment module, used to judge the user's fatigue level based on the monitored bad driving behavior; the fatigue level judgment module is integrated in the fatigue monitoring module or the in-vehicle infotainment module; the in-vehicle infotainment module is connected to the fatigue monitoring module, and is used to trigger the corresponding voice interaction module according to the fatigue level; this patent evaluates the driver's status and combines the in-vehicle infotainment module to relieve the driver's fatigue, but using car entertainment to relieve the driver's fatigue, such as playing music, cannot improve the driver's visual fatigue. If playing videos is used, it will distract the driver and also affect driving safety.

[0006] To this end, the present invention proposes a method for in-vehicle information interaction based on voice recognition, a vehicle system and a vehicle. Summary of the invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a method for in-vehicle information interaction based on voice recognition, a vehicle system and a vehicle, which introduce environmental factors to intelligently evaluate in-vehicle voice commands, generate intelligent interactive commands to control the vehicle, and improve driving safety.

[0008] To achieve the above-mentioned purpose, the present invention proposes a method for in-vehicle information interaction based on voice recognition, a vehicle system and a vehicle. The method for in-vehicle information interaction based on voice recognition includes:

[0009] Collecting sound training data, and training a timbre recognition model for recognizing timbre categories based on the sound training data;

[0010] Acquire in-vehicle sound data, divide the sound data into frames to obtain m groups of framed audio, where m is an integer greater than 1, extract voice commands from the sound data, and input the framed audio corresponding to the voice commands into a timbre recognition model to output a timbre category;

[0011] Acquire environmental data in real time and generate intelligent interactive instructions based on voice commands, timbre categories and environmental data.

[0012] Preferably, the environmental data includes the vehicle driving state, the opening and closing state of the wipers and windows, and the vehicle driving state includes the vehicle speed;

[0013] The method for generating intelligent interaction instructions based on voice instructions, timbre categories and environmental data includes:

[0014] Extracting the vehicle speed of the vehicle driving state in the environmental data, and calculating a comprehensive score based on the timbre category and the vehicle speed;

[0015] Compare the comprehensive score with the preset comprehensive score threshold to determine whether to generate an initial instruction;

[0016] If the comprehensive score is greater than or equal to the preset comprehensive score threshold, an initial instruction is generated;

[0017] If the comprehensive score is less than the preset comprehensive score threshold, no initial instruction is generated;

[0018] Calculate based on the initial instructions and generate intelligent interaction instructions.

[0019] Preferably, the training method of the timbre recognition model is:

[0020] Collect multiple groups of sound data in advance, detect each group of sound data to obtain each group of timbre feature data, and generate corresponding actual labels for each group of timbre feature data according to the sound data category when collected. When the sound data category is elderly, the actual label is 2, when the sound data category is adult, the actual label is 1, and when the sound data category is child, the actual label is 0;

[0021] Each set of timbre feature data is used as input of a second machine learning model, wherein the second machine learning model uses the predicted labeling of each set of timbre feature data as output, the actual labeling as a prediction target, and minimizing the sum of the prediction accuracies of all timbre feature data as a training target; the second machine learning model is trained until the sum of the prediction accuracies reaches convergence, the training is stopped, and the trained second machine learning model is used as a timbre recognition model;

[0022] The second machine learning model is a naive Bayes model or a support vector machine model.

[0023] Preferably, it also includes:

[0024] Collect facial training data, and train a state recognition model for identifying abnormal driving states based on the facial training data;

[0025] Acquire the facial data of the driver in the car in real time, input the facial data into the trained state recognition model, and output the driver's state;

[0026] Active interaction instructions are generated based on the driver status and environmental data, and the vehicle system controls the vehicle based on the active interaction instructions.

[0027] Preferably, the facial training data includes facial video data of the driver in normal driving state and abnormal driving state;

[0028] The facial video data of abnormal driving state includes the facial video data of artificially simulating dangerous driving;

[0029] The facial video data in a normal driving state includes facial video data artificially simulating normal driving.

[0030] Preferably, the training process of the state recognition model is as follows:

[0031] The label corresponding to the normal driving state in the facial training data is set to 0, and the label corresponding to the abnormal driving state in the facial training data is set to 1, and the facial training data and the corresponding labels are constructed as a data set of the first machine learning model; the data set is divided into a training set, a validation set and a test set, wherein the training set accounts for 70% of the data set, and the validation set and the test set each account for 15% of the data set;

[0032] The training set is used as the input of a first machine learning model, and the first machine learning model takes the probability that the driver is in an abnormal driving state as the output; the probability that the driver is in an abnormal driving state in the facial video data captured in real time by the in-car camera is used as the prediction target, and minimizing the loss function value of the first machine learning model is used as the training target; when the loss function value of the first machine learning model is less than or equal to the preset target loss value, the training is stopped;

[0033] The first machine learning model is a two-stream convolutional neural network model or a 3D convolutional neural network model;

[0034] An abnormal driving state probability threshold is set. When the probability that the driver is in an abnormal driving state is greater than or equal to the abnormal driving state probability threshold, the driver is marked as driving dangerously.

[0035] Preferably, the method for generating active interaction instructions based on driver status and environmental data includes:

[0036] If the driver is marked as dangerous driving and the vehicle is in motion, an active interaction instruction is generated;

[0037] If the driver marks it as normal driving and the vehicle is in driving state, no active interaction command will be generated;

[0038] Active interactive commands include turning on wipers and opening and closing windows at irregular intervals.

[0039] Preferably, the abnormal feature frequency in the facial data under abnormal driving state is obtained, the abnormal features include lowering the head or drooping eyelids, and the windshield wipers or windows are opened and closed based on the abnormal feature frequency.

[0040] Preferably, the active interaction instruction further includes dynamically adjusting the steering wheel power assist, and the method for dynamically adjusting the steering wheel power assist includes:

[0041] Calculate fatigue based on facial data;

[0042] The steering wheel assist coefficient is calculated based on the fatigue degree, and the vehicle steering system is controlled based on the steering wheel assist coefficient.

[0043] Preferably, the calculation expression of fatigue degree is:

[0044] ,in, For fatigue, is the eye movement characteristic index, is the time factor index; and is the weight of eye movement characteristics and time factors; and They are and The maximum value of .

[0045] Preferably, the calculation expression of the steering wheel power assistance coefficient is:

[0046] ,in is the steering wheel assist coefficient, is the basic assistance coefficient, To assist in adjusting the coefficient.

[0047] A vehicle computer system, used to implement the above-mentioned method for in-vehicle information interaction based on voice recognition, comprising:

[0048] A first acquisition module, used to collect facial training data;

[0049] A model training module, used for training a state recognition model for recognizing a driver's state based on facial training data;

[0050] The second acquisition module is used to collect facial data in real time;

[0051] The third collection module is used to collect environmental data;

[0052] The fourth collection module is used to collect the voice data of the speaker in the car;

[0053] A timbre recognition module is used to input sound data into a trained timbre recognition model and output a timbre category;

[0054] An active interaction module, used to generate active interaction instructions based on environmental data and abnormal driving status, and control the vehicle based on the active interaction instructions;

[0055] The intelligent interaction module is used to generate intelligent interaction instructions based on the tone category and environmental data, and control the vehicle based on the intelligent interaction instructions.

[0056] A vehicle comprises the vehicle system as described above.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The present invention collects the driver's facial data, in-car voice information and vehicle environmental data, and after extracting the voice commands in the car, it will evaluate the rationality of the voice commands in combination with the environmental data, determine whether to generate initial commands, calculate and generate intelligent interaction commands that meet the current environmental data based on the initial commands, and then control the vehicle based on the intelligent interaction commands, thereby improving the safety and intelligence of information interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a method for in-vehicle information interaction based on voice recognition provided in Embodiment 1 of the present invention;

[0060] Figure 2 A flowchart of a method for in-vehicle information interaction based on voice recognition provided in Embodiment 2 of the present invention;

[0061] Figure 3 A system structure diagram of a vehicle system provided by Embodiment 3 of the present invention;

[0062] Figure 4 A schematic diagram of a vehicle provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Example 1

[0065] See also Figure 1 As shown, the present invention provides an embodiment, a method for in-vehicle information interaction based on voice recognition, which specifically includes the following steps: collecting sound training data, and training a timbre recognition model for identifying timbre categories based on the sound training data; the training method of the timbre recognition model is: obtaining multiple groups of sound data in an experimental environment, detecting each group of sound data to obtain each group of timbre feature data, and generating corresponding actual annotations for each group of timbre feature data according to the sound data category when collected, when the sound data category is for the elderly, the actual annotation is 2, when the sound data category is for adults, the actual annotation is 1, and when the sound data category is for children, the actual annotation is 0; each group of timbre feature data is used as input of a second machine learning model, the second machine learning model uses the predicted annotation of each group of timbre feature data as output, the actual annotation as the prediction target, and minimizing the sum of the prediction accuracy of all timbre feature data as the training target; wherein the calculation formula for the prediction accuracy is: ,in, is the number of the timbre characteristic data, For prediction accuracy, For the The predicted annotations corresponding to the group timbre feature data, For the The actual annotations corresponding to the group of timbre feature data are obtained; the second machine learning model is trained until the sum of the prediction accuracies reaches convergence, the training is stopped and the trained second machine learning model is used as the timbre recognition model, the second machine learning model is a naive Bayes model or a support vector machine model; it should be noted that the convergence standard is determined by technical personnel in this field according to actual conditions and is not specifically limited here.

[0066] Acquire the in-vehicle sound data, divide the sound data into frames to obtain m groups of framed audio, where m is an integer greater than 1, extract the voice command in the sound data, and input the framed audio corresponding to the voice command into the timbre recognition model to output the timbre category, that is, in this embodiment, it is only necessary to determine the timbre of the sound data of the voice command issued in the vehicle; it should be noted that when dividing the sound data into frames, the required frame length and frame shift are determined by those skilled in the art according to the actual situation, and will not be elaborated on here;

[0067] Acquire environmental data in real time, the environmental data includes vehicle driving status, audio information, and wiper and window opening and closing status, the vehicle driving status includes vehicle speed, generate intelligent interaction commands based on voice commands, timbre categories and environmental data, and the vehicle system controls the vehicle based on the intelligent interaction commands; illustratively, if the keyword "open window" is obtained in the sound data, then combine it with the environmental data to evaluate whether the window opening command is executed, the window opening command includes the window opening position and opening size, such as when the vehicle is driving at high speed, the window opening size is evaluated as a small window opening; the purpose is to give a comprehensive evaluation of the initial voice command in combination with the environmental data inside or outside the vehicle, and to make intelligent corrections to the voice commands issued by the people in the vehicle.

[0068] Specifically, the method for generating intelligent interaction instructions based on voice instructions, timbre categories and environmental data includes:

[0069] Extracting the vehicle speed of the vehicle driving state in the environmental data, calculating a comprehensive score based on the timbre category and the vehicle speed, comparing the comprehensive score with a preset comprehensive score threshold, and determining whether to generate an initial instruction;

[0070] If the comprehensive score is greater than or equal to the preset comprehensive score threshold, an initial instruction is generated; if the comprehensive score is less than the preset comprehensive score threshold, no initial instruction is generated;

[0071] Calculate based on the initial instructions and generate intelligent interaction instructions.

[0072] More specifically, the decision of comprehensive scoring takes the voice command of window opening as an example, and the calculation expression of window opening decision is:

[0073] , where S is the comprehensive score, and are the weights of vehicle speed and age factors, respectively, and F is the standardized value of each factor (normalized between 0 and 1). is the vehicle speed, To identify the speaker's age, the value ranges from 0 to 1, such as 0.8 for the elderly or children and 0.5 for adults. The weights are adjusted by technicians in this field based on actual experience or feedback results;

[0074] A preset comprehensive score threshold is determined by those skilled in the art based on actual experience or actual needs and is not specifically limited herein. S is compared with the preset comprehensive score threshold to determine whether to generate an initial instruction. If S is greater than or equal to the preset comprehensive score threshold, an initial instruction is generated. If S is less than the preset comprehensive score threshold, no initial instruction is generated and a warning is given. The warning method includes a voice warning, such as playing a warning word through voice, such as "The current vehicle speed is too fast. There is a safety hazard if a child in the car issues a window opening instruction. Please ask the guardian to identify" etc.;

[0075] Specifically, the timbre category is obtained based on sound data. The sound data is input into a trained timbre recognition model to output the timbre category. The credibility of the command can be determined by judging the timbre category. For example, if a child is naughty and gives commands at will, some commands are not suitable for execution in a high-speed vehicle.

[0076] More specifically, if the initial command is determined to be executed, the initial command is intelligently corrected, such as the correction of the window opening voice command. The intelligent intervention calculation expression of the window opening voice command is: ,in, is the final window opening size, V is the current vehicle speed, The window opening and closing size of the initial command, is the influence coefficient of vehicle speed on the window opening size, It is determined by those skilled in the art based on experimental data.

[0077] The purpose is to avoid safety hazards caused by unreasonable instructions, such as children issuing window-opening instructions in a high-speed vehicle, by judging and intelligently correcting the initial instructions. For example, short instructions issued by people with speech impairments can be reasonably executed through intelligent correction, thus improving the interactive experience.

[0078] It is worth mentioning that in this embodiment, S is a calculation expression for the comprehensive score, taking windowing as an example, where S is a relevant factor of the comprehensive score, and those skilled in the art can incorporate more and make different judgments for different types of instructions.

[0079] Example 2

[0080] Based on the above embodiments, see Figure 2 As shown, the method for in-vehicle information interaction based on voice recognition also includes:

[0081] Collecting facial training data, the facial training data includes facial video data of the driver in normal driving state and abnormal driving state; the facial video data of abnormal driving state includes facial video data of artificially simulated dangerous driving, that is, in an experimental environment, collecting facial video data of people with different face shapes when they are driving dangerously, such as drooping eyelids or frequently lowering their heads; the facial video data of normal driving state includes facial video data of artificially simulated normal driving, and training a state recognition model for identifying abnormal driving state based on the facial training data;

[0082] Acquire the facial data of the driver in the car in real time, input the facial data into the trained state recognition model, and output the driver's state;

[0083] Active interaction instructions are generated based on the driver's state and environmental data, and the vehicle system controls the vehicle based on the active interaction instructions; specifically, if the driver is marked as dangerous driving and the vehicle is in a driving state, an active interaction instruction is generated; the active interaction instruction includes irregularly turning on the wipers or opening and closing the windows;

[0084] For example, when it is found that the driver is in a bad state, the vehicle is in a driving state, the wipers are not turned on, and the windows are closed, an active interaction instruction is generated, and the active interaction instruction includes irregularly turning on the wipers, playing music or prompting content audio, opening the windows, or turning on the air conditioning external circulation;

[0085] The purpose is to relieve the driver's visual fatigue by turning on the wipers irregularly, play music or prompt audio to relax the driver and reduce mental fatigue, open the windows or turn on the air conditioning external circulation to refresh the air in the car and relieve the driver's fatigue.

[0086] Specifically, the training process of the state recognition model is as follows:

[0087] The label corresponding to the normal driving state in the facial training data is set to 0, and the label corresponding to the abnormal driving state in the facial training data is set to 1, and the facial training data and the corresponding labels are constructed as a data set of the first machine learning model; the data set is divided into a training set, a validation set and a test set, wherein the training set accounts for 70% of the data set, and the validation set and the test set each account for 15% of the data set;

[0088] The training set is used as the input of a first machine learning model, and the first machine learning model takes the probability that the driver is in an abnormal driving state as the output; the probability that the driver is in an abnormal driving state in the facial video data captured in real time by the in-car camera is used as the prediction target, and minimizing the loss function value of the first machine learning model is used as the training target; when the loss function value of the first machine learning model is less than or equal to the preset target loss value, the training is stopped;

[0089] The first machine learning model loss function can be mean square error (MSE) or cross entropy (CE);

[0090] For example, the mean square error (MSE) is calculated by dividing the loss function value by The model is trained with minimization as the goal, so that the first machine learning model better fits the data, thereby improving performance and accuracy; i is the facial training data group number; is the number of facial training data sets; is the label corresponding to the i-th group of facial training data, is the predicted label based on the i-th group of facial training data.

[0091] The first machine learning model can be a two-stream convolutional neural network model or a 3D convolutional neural network model; other model parameters of the first machine learning model, such as the depth of the network model, the number of neurons in each layer, the activation function used by the network model, and the optimization of the loss function, are all achieved through actual engineering and obtained through continuous experimental tuning.

[0092] Set an abnormal driving state probability threshold. When the probability of the driver being in an abnormal driving state is greater than or equal to the abnormal driving state probability threshold, the driver is marked as driving dangerously. The abnormal driving state probability threshold is set by the staff based on the actual operation results of the first machine learning model. If the first machine learning model runs well and has a high recognition accuracy, the abnormal driving state probability threshold can be set lower to make the recognition ability more sensitive. If the recognition effect of the first machine learning model is average, the abnormal driving state probability threshold is set higher to prevent misidentification. For example, if the recognition accuracy of the first machine learning model is high, the abnormal driving state probability threshold is set to 70%. If the recognition accuracy of the first machine learning model is average, the abnormal driving state probability threshold is set to 90%.

[0093] Furthermore, the abnormal feature frequency in the facial data under abnormal driving status is obtained, the abnormal features include lowering the head or drooping eyelids, and the wipers or windows are turned on and off based on the abnormal feature frequency. The purpose is to obtain the driver's fatigue state in advance. For example, according to the driver's eyelid drooping frequency, the driver's fatigue state can be matched by the wiper opening frequency, so that the wipers can be turned on the next time the driver's eyelids droop to alert the driver.

[0094] Furthermore, the active interaction instruction also includes dynamically adjusting the steering wheel power assistance, and the method of dynamically adjusting the steering wheel power assistance includes:

[0095] Calculate fatigue based on facial data;

[0096] The steering wheel assist coefficient is calculated based on the fatigue degree, and the vehicle steering system is controlled based on the steering wheel assist coefficient.

[0097] Specifically, the calculation expression of fatigue is: ,in, For fatigue, It is an eye movement feature index (such as blink frequency or eye movement speed) extracted from facial video data. The specific extraction method will not be elaborated in detail here. It is a time factor index (such as continuous driving time), provided by the on-board computer or the continuous collection time of facial video data; and is the weight of eye movement characteristics and time factors, and The weights are assigned by technicians in this field based on actual application scenarios and importance, and the sum of the weights is 1; and are the maximum values ​​of each indicator, used for standardization and , technical personnel in this field need to make adjustments based on actual data to ensure that each indicator value can correctly reflect the degree of fatigue.

[0098] More specifically, the steering wheel power assist is dynamically adjusted based on the fatigue level. The power assist dynamic adjustment method is: the steering wheel power assist coefficient is calculated based on the fatigue level, and the steering wheel power assist coefficient is used as the adjustment parameter of the steering wheel power assist. The calculation expression of the steering wheel power assist coefficient is: ,in is the steering wheel assist coefficient, Is the basic power-assistance coefficient, provided by the basic vehicle data; It is the power assist adjustment coefficient, which is used to control the sensitivity of power assist changes. It can be adjusted according to vehicle characteristics and driver feedback, and is not specifically limited here.

[0099] Furthermore, an interactive method for preventing driver fatigue based on fatigue and dynamic adjustment of steering wheel assistance includes: real-time monitoring of the driver's fatigue, calculating the current steering wheel assistance coefficient, controlling the steering system, adjusting the steering wheel assistance according to the calculated value, repeating the above steps, real-time monitoring and adjusting the assistance, and if the driver is out of fatigue, stopping the information interaction of the steering wheel assistance adjustment.

[0100] The purpose of this embodiment is to analyze the driver's fatigue state and dynamically adjust the steering wheel power coefficient to achieve the purpose of reminding the driver. This method will not interfere with the driver's vision and can keep the driver focused at all times.

[0101] Example 3

[0102] Based on the above embodiments, see Figure 3As shown, a vehicle system is used to implement the above-mentioned method of in-vehicle information interaction based on voice recognition, including a first acquisition module, a model training module, a second acquisition module, a third acquisition module, a fourth acquisition module, a timbre recognition module, an active interaction module and an intelligent interaction module, wherein each module is connected through a wired and / or wireless network;

[0103] The first acquisition module is used to collect facial training data, that is, technical personnel in this field collect facial training data through the big data cloud. Data encryption is only provided to the model training module for model training to avoid information leakage.

[0104] The model training module is used to train a state recognition model for identifying the driver's state based on facial training data. The training process of the state recognition model is as follows:

[0105] The label corresponding to the normal driving state in the facial training data is set to 0, and the label corresponding to the abnormal driving state in the facial training data is set to 1, and the facial training data and the corresponding labels are constructed as a data set of the first machine learning model; the data set is divided into a training set, a validation set and a test set, wherein the training set accounts for 70% of the data set, and the validation set and the test set each account for 15% of the data set;

[0106] The training set is used as the input of a first machine learning model, and the first machine learning model takes the probability that the driver is in an abnormal driving state as the output; the probability that the driver is in an abnormal driving state in the facial video data captured in real time by the in-car camera is used as the prediction target, and minimizing the loss function value of the first machine learning model is used as the training target; when the loss function value of the first machine learning model is less than or equal to the preset target loss value, the training is stopped;

[0107] The first machine learning model loss function can be mean square error (MSE) or cross entropy (CE);

[0108] For example, the mean square error (MSE) is calculated by dividing the loss function value by The model is trained with minimization as the goal, so that the first machine learning model better fits the data, thereby improving performance and accuracy; i is the facial training data group number; is the number of facial training data sets; is the label corresponding to the i-th group of facial training data, is the predicted label based on the i-th group of facial training data.

[0109] The first machine learning model can be a two-stream convolutional neural network model or a 3D convolutional neural network model; other model parameters of the first machine learning model, such as the depth of the network model, the number of neurons in each layer, the activation function used by the network model, and the optimization of the loss function, are all achieved through actual engineering and obtained through continuous experimental tuning.

[0110] Set an abnormal driving state probability threshold. When the probability of the driver being in an abnormal driving state is greater than or equal to the abnormal driving state probability threshold, the driver is marked as driving dangerously. The abnormal driving state probability threshold is set by the staff based on the actual operation results of the first machine learning model. If the first machine learning model runs well and has a high recognition accuracy, the abnormal driving state probability threshold can be set lower to make the recognition ability more sensitive. If the recognition effect of the first machine learning model is average, the abnormal driving state probability threshold is set higher to prevent misidentification. For example, if the recognition accuracy of the first machine learning model is high, the abnormal driving state probability threshold is set to 70%. If the recognition accuracy of the first machine learning model is average, the abnormal driving state probability threshold is set to 90%.

[0111] The second acquisition module is used to collect facial data in real time. In this embodiment, the second acquisition module uses a high-definition camera. The high-definition camera can be installed on the windshield inside the car, and the detection part is facing the driver's face.

[0112] The third acquisition module is used to collect environmental data. The third acquisition module of this implementation includes multiple acquisition units, and the multiple acquisition units are distributed on the vehicle, including collecting vehicle speed, window and wiper opening and closing status, and vehicle inside and outside temperature. The third acquisition module of this implementation may directly interact with the on-board computer for data, and monitor driving parameters such as vehicle speed by acquiring data from the on-board computer.

[0113] The fourth acquisition module is used to collect the voice data of the speaker in the car. In this embodiment, the fourth acquisition module adopts a microphone to collect the voice data of the speaker in the car through the microphone.

[0114] The timbre recognition module is used to input the sound data into the trained timbre recognition model and output the timbre category. The training method of the timbre recognition model is as follows: multiple groups of sound data are obtained in an experimental environment, each group of sound data is detected to obtain each group of timbre feature data, and each group of timbre feature data is generated according to the sound data category when collected. The actual label is 2 when the sound data category is elderly, 1 when the sound data category is adult, and 0 when the sound data category is child; each group of timbre feature data is used as the input of the second machine learning model, and the second machine learning model uses the predicted labeling of each group of timbre feature data as the output, the actual labeling as the prediction target, and the minimization of the sum of the prediction accuracy of all timbre feature data as the training target; wherein the calculation formula for the prediction accuracy is: ,in, is the number of the timbre characteristic data, For prediction accuracy, For the The predicted annotations corresponding to the group timbre feature data, For the The actual annotations corresponding to the group of timbre feature data are obtained; the second machine learning model is trained until the sum of the prediction accuracies reaches convergence, the training is stopped and the trained second machine learning model is used as the timbre recognition model, the second machine learning model is a naive Bayes model or a support vector machine model. It should be noted that the convergence standard is determined by technical personnel in this field according to actual conditions and is not specifically limited here.

[0115] The active interaction module is used to generate active interaction instructions based on environmental data and abnormal driving conditions, and control the vehicle based on the active interaction instructions; the active interaction instructions include dynamically adjusting the steering wheel assist; the method for dynamically adjusting the steering wheel assist includes: calculating fatigue based on facial data; calculating the steering wheel assist coefficient based on fatigue, and controlling the vehicle steering system based on the steering wheel assist coefficient; it is worth mentioning that the active interaction module requires the vehicle system to interact with some controllers in the vehicle. The specific interaction protocol and control are set by technical personnel in this field according to the actual vehicle model, and different vehicle models can also be matched using data coupling.

[0116] The intelligent interaction module is used to generate intelligent interaction instructions based on the timbre category and the environmental data, and control the vehicle based on the intelligent interaction instructions; the method for generating intelligent interaction instructions based on voice instructions and environmental data includes:

[0117] Input the sound data into the trained timbre recognition model and output the timbre category;

[0118] Extracting the vehicle speed of the vehicle driving state in the environmental data, calculating a comprehensive score based on the timbre category and the vehicle speed, comparing the comprehensive score with a preset comprehensive score threshold, and determining whether to generate an initial instruction;

[0119] If the comprehensive score is greater than or equal to the preset comprehensive score threshold, an initial instruction is generated; if the comprehensive score is less than the preset comprehensive score threshold, no initial instruction is generated;

[0120] Calculate based on the initial instructions and generate intelligent interaction instructions.

[0121] See also Figure 4 As shown, a vehicle includes the vehicle system as described above.

[0122] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0123] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0124] The above preset parameters or preset thresholds are all set by technicians in this field according to actual conditions or obtained through large-scale data simulation.

[0125] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A method for in-vehicle information interaction based on voice recognition, characterized in that: include: Collecting sound training data, and training a timbre recognition model for recognizing timbre categories based on the sound training data; Acquire in-vehicle sound data, divide the sound data into frames to obtain m groups of framed audio, where m is an integer greater than 1, extract voice commands from the sound data, and input the framed audio corresponding to the voice commands into a timbre recognition model to output a timbre category; Acquire environmental data in real time and generate intelligent interactive instructions based on voice commands, timbre categories and environmental data.

2. The method for in-vehicle information interaction based on voice recognition according to claim 1, characterized in that: Environmental data includes vehicle driving status, windshield wiper and window opening and closing status, and vehicle driving status includes vehicle speed; The method for generating intelligent interaction instructions based on voice instructions, timbre categories and environmental data includes: Extracting the vehicle speed of the vehicle driving state in the environmental data, and calculating a comprehensive score based on the timbre category and the vehicle speed; Compare the comprehensive score with the preset comprehensive score threshold to determine whether to generate an initial instruction; If the comprehensive score is greater than or equal to the preset comprehensive score threshold, an initial instruction is generated; If the comprehensive score is less than the preset comprehensive score threshold, no initial instruction is generated; Calculate based on the initial instructions and generate intelligent interaction instructions.

3. The method for in-vehicle information interaction based on voice recognition according to claim 1, characterized in that: The training method of the timbre recognition model is: Collect multiple groups of sound data in advance, detect each group of sound data to obtain each group of timbre feature data, and generate corresponding actual labels for each group of timbre feature data according to the sound data category when collected. When the sound data category is elderly, the actual label is 2, when the sound data category is adult, the actual label is 1, and when the sound data category is child, the actual label is 0; Each set of timbre feature data is used as input of a second machine learning model, wherein the second machine learning model uses the predicted labeling of each set of timbre feature data as output, the actual labeling as a prediction target, and minimizing the sum of the prediction accuracies of all timbre feature data as a training target; the second machine learning model is trained until the sum of the prediction accuracies reaches convergence, the training is stopped, and the trained second machine learning model is used as a timbre recognition model; The second machine learning model is a naive Bayes model or a support vector machine model.

4. The method for in-vehicle information interaction based on voice recognition according to claim 1, characterized in that: Also includes: Collect facial training data, and train a state recognition model for identifying abnormal driving states based on the facial training data; Acquire the facial data of the driver in the car in real time, input the facial data into the trained state recognition model, and output the driver's state; Active interaction instructions are generated based on the driver status and environmental data, and the vehicle system controls the vehicle based on the active interaction instructions.

5. The method for in-vehicle information interaction based on voice recognition according to claim 4, characterized in that: The facial training data includes facial video data of the driver in normal driving state and abnormal driving state; The facial video data of abnormal driving state includes the facial video data of artificially simulating dangerous driving; The facial video data in a normal driving state includes facial video data artificially simulating normal driving.

6. The method for in-vehicle information interaction based on voice recognition according to claim 4, characterized in that: The training process of the state recognition model is as follows: The label corresponding to the normal driving state in the facial training data is set to 0, and the label corresponding to the abnormal driving state in the facial training data is set to 1, and the facial training data and the corresponding labels are constructed as a data set of the first machine learning model; the data set is divided into a training set, a validation set and a test set, wherein the training set accounts for 70% of the data set, and the validation set and the test set each account for 15% of the data set; Using the training set as input to a first machine learning model, wherein the first machine learning model uses the probability that the driver is in an abnormal driving state as output; The probability of the driver being in an abnormal driving state in the facial video data captured by the in-car camera in real time is used as the prediction target, and minimizing the loss function value of the first machine learning model is used as the training target; when the loss function value of the first machine learning model is less than or equal to the preset target loss value, the training is stopped; The first machine learning model is a two-stream convolutional neural network model or a 3D convolutional neural network model; An abnormal driving state probability threshold is set. When the probability that the driver is in an abnormal driving state is greater than or equal to the abnormal driving state probability threshold, the driver is marked as driving dangerously.

7. The method for in-vehicle information interaction based on voice recognition according to claim 4, characterized in that: The method for generating proactive interaction instructions based on driver status and environmental data includes: If the driver is marked as dangerous driving and the vehicle is in motion, an active interaction instruction is generated; If the driver marks it as normal driving and the vehicle is in driving state, no active interaction command will be generated; Active interactive commands include turning on wipers and opening and closing windows at irregular intervals.

8. The method for in-vehicle information interaction based on voice recognition according to claim 7, characterized in that: Obtain the abnormal feature frequency in the facial data under abnormal driving conditions. Abnormal features include lowering the head or drooping eyelids. Open or close the wipers or windows based on the abnormal feature frequency.

9. The method for in-vehicle information interaction based on voice recognition according to claim 8, characterized in that: The active interaction instruction also includes dynamically adjusting the steering wheel power assist. The method of dynamically adjusting the steering wheel power assist includes: Calculate fatigue based on facial data; The steering wheel assist coefficient is calculated based on the fatigue degree, and the vehicle steering system is controlled based on the steering wheel assist coefficient.

10. The method for in-vehicle information interaction based on voice recognition according to claim 9, characterized in that: The calculation expression of fatigue degree is: ,in, For fatigue, is the eye movement characteristic index, is the time factor index; and is the weight of eye movement characteristics and time factors; and They are and The maximum value of .

11. The method for in-vehicle information interaction based on voice recognition according to claim 10, characterized in that: The calculation expression of the steering wheel power assist coefficient is: ,in is the steering wheel assist coefficient, is the basic assistance coefficient, To assist in adjusting the coefficient.

12. A vehicle computer system, used to implement the method for in-vehicle information interaction based on voice recognition according to any one of claims 1 to 11, characterized in that: include: A first acquisition module, used to collect facial training data; A model training module, used for training a state recognition model for recognizing a driver's state based on facial training data; The second acquisition module is used to collect facial data in real time; The third collection module is used to collect environmental data; The fourth collection module is used to collect the voice data of the speaker in the car; A timbre recognition module is used to input sound data into a trained timbre recognition model and output a timbre category; An active interaction module, used to generate active interaction instructions based on environmental data and abnormal driving status, and control the vehicle based on the active interaction instructions; The intelligent interaction module is used to generate intelligent interaction instructions based on the tone category and environmental data, and control the vehicle based on the intelligent interaction instructions.

13. A vehicle, characterized in that: It includes the vehicle system as claimed in claim 12.

Images