Driver state determination method and device, electronic equipment and storage medium

Abstract

The invention discloses a driver state determination method and device, electronic equipment and a storage medium, and relates to the technical field of vehicle detection. The method comprises the steps of determining a target micro-expression type of a driver in a vehicle at a current moment based on collected target facial image data of the driver in the vehicle and a trained micro-expression recognition model; based on a preset natural language processing algorithm, performing voice processing on the collected target voice stream data of the driver in the vehicle, and determining target semantics and target emotion types of the driver in the vehicle at the current moment; determining a target emotion type of the driver in the vehicle at the current moment based on the target micro-expression type, the target semantics and the target emotion type; and determining a target psychological state of the driver in the vehicle at the current moment based on the target emotion type and a preset emotion psychological state mapping table. According to the method, the psychological state of the driver in the vehicle can be analyzed and dredged, so that the accuracy of the monitoring result of state monitoring of the driver is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle detection technology, and in particular to a driver status determination method, device, electronic device, and storage medium. Background Art

[0002] At present, with the popularization of automobiles, driving safety has become the focus of social attention. The driver's psychological state has a significant impact on driving safety. Negative emotions such as fatigue, anxiety, and anger can significantly reduce the driver's reaction ability and judgment, increasing the risk of traffic accidents. Some traditional vehicle-mounted systems can analyze the visual data collected by cameras to monitor driver fatigue.

[0003] However, the relatively single reliance on visual data may be affected by factors such as the driver wearing glasses and facial occlusion. In addition, the single visual data lacks in-depth analysis of the driver's overall psychological state and effective intervention measures, resulting in inaccurate monitoring results for the driver's status. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, electronic device and storage medium for determining a driver's state. The embodiments provided by the present application solve the technical problem in the prior art of lacking in-depth analysis of the driver's comprehensive psychological state and effective intervention measures, resulting in inaccurate monitoring results of the driver's state monitoring. The embodiments provided by the present application can conduct in-depth analysis and effective guidance and intervention on the psychological state of the driver in the vehicle, thereby improving the accuracy of the monitoring results of the driver's state monitoring.

[0005] According to a first aspect of the embodiments of the present application, a method for determining a driver's state is provided, the method comprising:

[0006] Determining a target micro-expression type of the driver in the vehicle at a current moment based on the collected target facial image data of the driver in the vehicle and the trained micro-expression recognition model;

[0007] Based on a preset natural language processing algorithm, speech processing is performed on the collected target speech stream data of the driver in the vehicle to determine the target semantics and target emotion type of the driver in the vehicle at the current moment;

[0008] Determining a target emotion type of the driver at the current moment based on the target micro-expression type, the target semantics, and the target emotion type;

[0009] Determining the target psychological state of the driver at the current moment based on the target emotion type and a preset emotion psychological state mapping table;

[0010] Based on the target psychological state, driving guidance information for providing psychological prompts and guidance to the driver in the vehicle is determined.

[0011] In a feasible implementation, performing speech processing on the collected target speech stream data of the driver in the vehicle based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the vehicle at the current moment includes:

[0012] Inputting the collected target voice stream data of the driver in the vehicle into a trained voice recognition model for voice recognition, and determining text data corresponding to the target voice stream data;

[0013] Performing semantic analysis on the text data based on the text data and a preset natural language processing algorithm to determine the target semantics of the driver in the vehicle at the current moment;

[0014] Based on the text data and the trained emotion classification model, a target emotion type of the driver in the vehicle at the current moment is determined.

[0015] In a feasible implementation, determining the target emotion type of the driver at the current moment based on the target micro-expression type, the target semantics, and the target emotion type includes:

[0016] Based on a trained preset multi-layer perceptual neural network model, feature fusion and emotion classification are performed on the target micro-expression type, the target semantics, and the target emotion type to determine the target emotion type of the driver in the car at the current moment.

[0017] In a feasible implementation manner, determining driving guidance information for providing psychological guidance to the driver in the vehicle based on the target psychological state includes:

[0018] Determining a target risk level corresponding to the target psychological state based on the target psychological state and a preset risk level assessment model;

[0019] The target psychological state and the target risk level are input into a preset large language model to determine driving guidance information for providing psychological prompts and guidance to the driver in the vehicle.

[0020] In a feasible implementation, inputting the target psychological state and the target risk level into a preset large language model to determine driving guidance information for providing psychological guidance to the driver in the vehicle includes:

[0021] If the target risk level is greater than a preset risk level, the target psychological state and the target risk level are input into a preset large language model, and the driving counseling information is output, where the driving counseling information includes psychological counseling information for psychological counseling of the driver in the vehicle and auxiliary driving information for providing auxiliary prompts to the driver in the vehicle; or

[0022] If the target risk level is less than or equal to the preset risk level, the target psychological state and the target risk level are input into a preset large language model, and the driving guidance information is output, where the driving guidance information includes the psychological guidance information.

[0023] In a feasible implementation manner, after inputting the target psychological state and the target risk level into a preset large language model to determine driving guidance information for providing psychological guidance to the driver in the vehicle, the determination method further includes:

[0024] Obtaining feedback behavior of the in-vehicle driver in response to the psychological counseling information and / or the assisted driving information;

[0025] Based on the feedback behavior, the preset large language model is optimized.

[0026] In a feasible embodiment, before determining the target micro-expression type of the driver in the vehicle at the current moment based on the collected target facial image data of the driver in the vehicle and the trained micro-expression recognition model, the determination method further includes:

[0027] Acquire initial facial image data and initial voice stream data of the driver in the car;

[0028] Performing grayscale processing and noise reduction processing on the initial facial image data to determine target facial image data of the driver in the vehicle;

[0029] Noise reduction and dereverberation processing are performed on the initial voice stream data to determine target voice stream data of the driver in the vehicle.

[0030] In a second aspect of the embodiments of the present application, an apparatus for determining a driver's state is provided.

[0031] The driver state determining device comprises:

[0032] a first determination module, configured to determine a target micro-expression type of the driver in the vehicle at a current moment based on collected target facial image data of the driver in the vehicle and a trained micro-expression recognition model, wherein the target facial image data is used to represent the preprocessed facial image data;

[0033] a second determination module, configured to perform speech processing on the collected target speech stream data of the driver in the vehicle based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the vehicle at the current moment, wherein the target speech stream data is used to represent the preprocessed speech stream data;

[0034] A third determination module is configured to determine a target emotion type of the driver at the current moment based on the target micro-expression type, the target semantics, and the target emotion type;

[0035] A fourth determining module is configured to determine the target psychological state of the driver at the current moment based on the target emotion type and a preset emotion psychological state mapping table;

[0036] The fifth determining module is used to determine driving guidance information for providing psychological prompts and guidance to the driver in the vehicle based on the target psychological state.

[0037] In a third aspect of an embodiment of the present application, an embodiment of the present application provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the above-mentioned driver status determination method.

[0038] In a fourth aspect of an embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is run by a processor, the steps of the driver status determination method as described above are executed.

[0039] The driver status determination method, device, electronic device and storage medium provided in the embodiments of the present application, compared with the existing technology, the embodiments provided by the present application determine the target micro-expression type of the driver in the car at the current moment based on the collected target facial image data of the driver in the car and the trained micro-expression recognition model, and then perform voice processing on the collected target voice stream data of the driver in the car based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the car at the current moment, and then determine the target emotion type of the driver in the car at the current moment based on the target micro-expression type, target semantics and target emotion type, and then determine the target psychological state of the driver in the car at the current moment based on the target emotion type and a preset emotion-psychological state mapping table, and finally determine driving guidance information for psychological prompts and guidance for the driver in the car based on the target psychological state. The embodiments provided by the present application can conduct in-depth analysis and effective guidance and intervention on the psychological state of the driver in the car, thereby improving the accuracy of the monitoring results of the driver's status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a method for determining a driver's state provided in an embodiment of the present application is shown;

[0041] Figure 2 A structural block diagram of a driver status determination device provided by an embodiment of the present application is shown;

[0042] Figure 3 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0043] Figure 2 and Figure 3 The corresponding relationship between the reference numerals and the names of the drawings is as follows:

[0044] 200 Driver state determination device; 210 First determination module; 220 Second determination module; 230 Third determination module; 240 Fourth determination module; 250 Fifth determination module; 300 Electronic device; 310 Processor; 320 Memory; 330 Bus. DETAILED DESCRIPTION

[0045] In order to better understand the technical solutions provided by the embodiments of this specification, the technical solutions of the embodiments of this specification are described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0046] In this article, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also include elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprising a ..." do not exclude the presence of other identical elements in the process, method, article or equipment comprising the elements. The term "two or more" includes two or more than two cases.

[0047] First, the application scenarios to which this application is applicable are introduced. The embodiments provided in this application are applicable to the field of vehicle detection technology, and in particular relate to a driver status determination method, device, electronic device and storage medium.

[0048] At present, the relatively single reliance on visual data may be affected by factors such as the driver wearing glasses and facial occlusion. In addition, the single visual data lacks in-depth analysis of the driver's comprehensive psychological state and effective intervention measures, resulting in inaccurate monitoring results for the driver's status monitoring.

[0049] Based on this, the embodiments of the present application provide a driver status determination method, device, electronic device and storage medium. The embodiments provided by the present application solve the technical problem that the existing technology lacks in-depth analysis of the driver's comprehensive psychological state and effective intervention measures, resulting in inaccurate monitoring results of the driver's status monitoring. The embodiments provided by the present application can conduct in-depth analysis and effective guidance and intervention on the psychological state of the driver in the vehicle, thereby improving the accuracy of the monitoring results of the driver's status monitoring.

[0050] Figure 1 FIG. 1 shows a flowchart of a method for determining a driver's state provided by an embodiment of the present application. Figure 1 As shown, the driver status determination method includes the following steps:

[0051] S101 : Determine the target micro-expression type of the driver in the car at the current moment based on the collected target facial image data of the driver in the car and the trained micro-expression recognition model.

[0052] In this step, when the embodiment provided by the present application wants to determine the driver's status, it is first necessary to collect the target facial image data of the driver in the car, and then input the collected target facial image data into the trained micro-expression recognition model for micro-expression recognition, and then determine the target micro-expression type of the driver in the car at the current moment.

[0053] It can be understood that the target facial image data acquisition device in the embodiments provided by this application can be customized and used according to different application scenarios and usage conditions. The embodiments provided by this application can use a driver monitoring system (DMS) camera to obtain target facial image data of the driver in the car, wherein the DMS camera needs to be installed in a suitable position in the car to capture the driver's facial image in real time and provide data support for micro-expression analysis. The DMS camera in the embodiments provided by this application captures a frame of target facial image every preset time (such as 1 second).

[0054] Among them, the trained micro-expression recognition model in the embodiment provided by the present application can be customized and used according to different application scenarios and usage conditions. The embodiment provided by the present application can use a convolutional neural network to extract features of the target facial image data (such as extracting feature vectors of facial micro-expressions through convolutional layers, pooling layers, and fully connected layers), and perform facial feature classification after feature extraction (such as classifying facial feature vectors based on at least the last fully connected layer, i.e., the classifier) ​​to identify the target micro-expression type of the driver in the car at the current moment, such as surprise, anger, sadness, and joy.

[0055] In the above, the embodiment provided by the present application recognizes the target micro-expressions of the driver in the car through a trained micro-expression recognition model. The trained micro-expression recognition model can capture subtle changes in facial expressions of the driver in the car, and even fleeting target micro-expressions can be accurately recognized and captured. That is, the trained micro-expression recognition model provides a key basis for analyzing the target psychological state of the driver in the car.

[0056] S102: Based on a preset natural language processing algorithm, the collected target voice stream data of the driver in the car is processed to determine the target semantics and target emotion type of the driver in the car at the current moment.

[0057] In this step, the embodiment provided by the present application needs to collect the target facial image data of the driver in the car at the same time as the target voice stream data of the driver in the car, and then perform voice processing on the collected target voice stream data (such as the content, tone and speaking speed of the driver in the car) through a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the car at the current moment, so as to assist in analyzing the target emotion type and target psychological state of the driver in the car.

[0058] It can be understood that the target facial image data acquisition device in the embodiments provided by the present application can be customized and used according to different application scenarios and usage conditions. The target facial image data acquisition device in the embodiments provided by the present application can be a microphone. The microphone in the embodiments provided by the present application continuously records the target voice stream data of the driver in the car.

[0059] The embodiment provided in this application comprehensively utilizes DMS cameras and microphones to simultaneously collect target facial image data and target voice stream data of the driver in the car, thereby achieving comprehensive monitoring of the driving status of the driver in the car from both visual and auditory dimensions. Compared with a single data source, it can more accurately and comprehensively reflect the target psychological state of the driver in the car, avoid misjudgment due to the limitations of a single data source, and improve the accuracy of the data.

[0060] Exemplarily, before obtaining the target facial image data and target voice stream data, the present application: obtains the initial facial image data and initial voice stream data of the driver in the car; performs grayscale processing and noise reduction processing on the initial facial image data to determine the target facial image data of the driver in the car; performs noise reduction processing and dereverberation processing on the initial voice stream data to determine the target voice stream data of the driver in the car.

[0061] It can be understood that the embodiment provided in the present application first obtains the initial facial image data of the driver in the car collected by the DMS camera and the initial voice stream data collected by the microphone, and then performs grayscale processing on the initial facial image data, converting the color initial facial image data into a grayscale image, and then enhances the contrast of the image through histogram equalization. Finally, Gaussian filtering is used to remove noise in the image to generate target facial image data of the driver in the car, which is used to improve the quality of the image data.

[0062] It should be noted that the embodiment provided in this application first obtains the initial voice stream data of the driver in the car collected by the microphone and performs voice preprocessing including noise reduction processing (adaptive filtering algorithm can be used for noise reduction processing to remove voice environment noise and background noise) and de-reverberation processing to determine the target voice stream data of the driver in the car to improve the quality of the voice data.

[0063] Exemplarily, based on a preset natural language processing algorithm, speech processing is performed on the collected target voice stream data of the driver in the car to determine the target semantics and target emotion type of the driver in the car at the current moment, including: inputting the collected target voice stream data of the driver in the car into a trained voice recognition model for speech recognition to determine the text data corresponding to the target voice stream data; based on the text data and a preset natural language processing algorithm, semantic analysis is performed on the text data to determine the target semantics of the driver in the car at the current moment; based on the text data and a trained emotion classification model, the target emotion type of the driver in the car at the current moment is determined.

[0064] It can be understood that the embodiment provided by the present application first inputs the collected target voice stream data of the driver in the car into a trained voice recognition model (such as a convolutional neural network, a recurrent neural network or a long short-term memory network), performs voice recognition on the target voice stream data, and converts the above target voice stream data into text data. Then, through a preset natural language processing algorithm, the converted text data is semantically analyzed, the sentences are decomposed into words or phrases, and the grammatical role of each unit (such as nouns, verbs, etc.) is marked to determine the target semantics of the driver in the car at the current moment.

[0065] At the same time, the embodiment provided by the present application analyzes the text data through a trained sentiment classification model to determine the target emotion type of the driver in the car at the current moment; the embodiment provided by the present application can also directly extract emotion type features from the target voice stream data, such as pitch changes, energy distribution, and pitch rhythm, and then use a classification algorithm to predict the target emotion type of the driver in the car at the current moment. This method does not require converting the target voice stream data into text, so it is more suitable for practical applications. For the above two methods of determining the target emotion type, one can be selected according to different application scenarios. By determining the target emotion type, the present application realizes the judgment of the emotional tendencies (such as positive, negative, and neutral emotions) and language intentions (key information) of the driver in the test.

[0066] S103: Determine the target emotion type of the driver at the current moment based on the target micro-expression type, target semantics, and target emotion type.

[0067] In this step, the embodiment provided in this application performs feature fusion on the target micro-expression type output by the trained micro-expression recognition model, the target semantics and target emotion type output by the preset natural language processing algorithm, and generates and determines the target emotion type that can better reflect the real in-car emotions of the driver.

[0068] It can be understood that the target emotion type in the embodiments provided in this application can be customized and used according to different application scenarios and usage conditions. The target emotion type in the embodiments provided in this application can specifically be fatigue, anger, etc.

[0069] Exemplarily, based on a trained preset multi-layer perception neural network model, feature fusion and emotion classification are performed on the target micro-expression type, target semantics, and target emotion type to determine the target emotion type of the driver in the car at the current moment.

[0070] It is understandable that the preset multi-layer perceptron neural network model in the embodiments provided in this application may be determined using a machine learning or deep learning method, such as:

[0071] Multilayer Perceptron (MLP) or decision tree, etc.

[0072] It should be noted that, in addition to using a pre-trained multi-layer perceptron neural network model for feature fusion and feature recognition, the embodiments provided in this application may also use:

[0073] Support Vector Machine (SVM), a supervised learning model, is used to fuse and determine features.

[0074] It can be understood that the embodiments provided in the present application can also perform weighted fusion of the emotional tendencies in the first emotional feature vector in the data of the target micro-expression type and the second emotional feature vector in the data of the target emotional type according to preset weight distribution rules to generate the target emotional type of the driver in the car at the current moment and the target emotional type data.

[0075] S104: Determine the target psychological state of the driver at the current moment based on the target emotion type and the preset emotion psychological state mapping table.

[0076] In this step, after determining the target emotion type, the embodiment provided by the present application needs to determine the target psychological state corresponding to the target emotion type. At this time, the embodiment provided by the present application can determine the target psychological state at the current moment corresponding to the above-mentioned preset emotion psychological state mapping table from the preset emotion psychological state mapping table, and map the weighted fusion result to the target psychological state of the driver in the car at the current moment based on the preset emotion psychological state mapping table, so that subsequent vehicles can provide accurate psychological counseling for the driver in the car based on the above-mentioned target psychological state, while enriching the understanding of the driver's psychological state and improving the user experience.

[0077] This application uses psychological algorithms (or preset emotional psychological state mapping tables) to comprehensively evaluate the psychological state of the driver in the car, such as fatigue, anxiety, anger, and concentration. At the same time, according to different target psychological states, a corresponding risk assessment model is established to provide a basis for subsequent decision-making.

[0078] S105: Determine driving guidance information for providing psychological prompts and guidance to the driver in the vehicle based on the target psychological state.

[0079] In this step, after determining the target psychological state, the embodiment provided by the present application needs to provide psychological prompts and guidance for the different target psychological states of the driver in the car, and determine the driving guidance information for the psychological prompts and guidance.

[0080] Exemplarily, the present application: determines the target risk level corresponding to the target psychological state based on the target psychological state and the preset risk level assessment model; inputs the target psychological state and the target risk level into the preset large language model to determine driving guidance information for psychological prompts and guidance for the driver in the car.

[0081] It can be understood that the embodiments provided in this application can determine the target risk level corresponding to the above-mentioned target psychological state based on the preset risk level assessment model, and through the preset large language model, more scientifically and accurately determine the driving guidance information for the driver in the car, so that the driver in the car can maintain a relatively calm state.

[0082] It should be noted that the preset risk level assessment model in the embodiments provided in this application is customized and used according to different application scenarios.

[0083] Exemplarily, if the target risk level is greater than the preset risk level, the target psychological state and the target risk level are input into the preset large language model, and driving guidance information is output, which includes psychological counseling information for psychological counseling of the driver in the vehicle and auxiliary driving information for providing auxiliary prompts to the driver in the vehicle; or if the target risk level is less than or equal to the preset risk level, the target psychological state and the target risk level are input into the preset large language model, and driving guidance information is output, which includes psychological counseling information.

[0084] It should be noted that, in the embodiment provided in the present application, after determining the target risk level corresponding to the target psychological state, the target risk level can be first judged, such as: anger corresponds to the first risk level, fatigue corresponds to the second risk level, and anxiety corresponds to the third risk level. The target psychological state and the target risk level are then input together into a preset large language model, and the type and content of the output driving guidance information are determined based on the size of the determined target risk level.

[0085] It can be understood that the embodiment provided by the present application generates an intervention strategy including soothing music playback instructions and speed limit prompts when the target psychological state at the current moment is characterized by a severe anger state greater than the preset risk level; when the target psychological state is characterized by a severe fatigue state greater than the preset risk level, psychological counseling information including voice warnings is generated, such as "You have been driving for a long time and may be a little tired. It is recommended that you find a safe place to rest" and rest stop navigation information for assisting the driver in the car, and other auxiliary driving information, such as "text prompts, icon warnings, voice, scene music, and dialogue, etc."

[0086] Among them, the embodiments provided by the present application can output soothing music or voice warnings to alleviate the target psychological state of the driver in the car through the car speakers, and display speed limit prompts or rest stop navigation information through the central control display screen. That is, the embodiments provided by the present application can use a preset large language model to provide psychological counseling for the different target psychological states exhibited by the driver in the car, realize personalized monitoring of the driver's status, and help alleviate the carelessness of the driver in the car due to fatigue driving or excessive excitement, so that the driver can feel the exclusive personalized psychological counseling service and auxiliary prompt service when encountering complex road conditions.

[0087] In the above, the embodiments provided by this application can provide psychological prompts and guidance to the driver in the car through voice or vision, so that the driver in the car can receive information without being distracted, thereby improving the user experience. In addition, the embodiments provided by this application can also be combined with more types of sensor detection data, such as heart rate monitoring and eye tracking, to further improve the monitoring of the target psychology of the driver in the car.

[0088] Illustratively, after determining the driving counseling information for providing psychological guidance to the driver in the car, the embodiment provided by the present application can also: obtain the feedback behavior of the driver in the car in response to the psychological counseling information and / or assisted driving information; and optimize the preset large language model based on the feedback behavior.

[0089] It is understandable that the embodiments provided in the present application can also record the feedback behavior of the driver in the car to the psychological counseling information and / or assisted driving information, and when it is determined that the driver in the car has not executed the psychological counseling within the preset time, the voice intensity level of the subsequent fatigue warning is increased. If the driver actively turns off the anger counseling prompt, the frequency of intervention in similar target emotional states is reduced, thereby reducing the risk of traffic accidents.

[0090] The driver status determination method provided in the embodiment of the present application, compared with the existing technology, the embodiment provided by the present application determines the target micro-expression type of the driver in the car at the current moment based on the collected target facial image data of the driver in the car and the trained micro-expression recognition model, and then performs voice processing on the collected target voice stream data of the driver in the car based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the car at the current moment, and then determines the target emotion type of the driver in the car at the current moment based on the target micro-expression type, target semantics and target emotion type, and then determines the target psychological state of the driver in the car at the current moment based on the target emotion type and a preset emotion-psychological state mapping table, and finally determines driving guidance information for psychological prompts and guidance for the driver in the car based on the target psychological state. The embodiment provided by the present application can conduct in-depth analysis and effective guidance and intervention on the psychological state of the driver in the car, thereby improving the accuracy of the monitoring results of the driver's status monitoring, further ensuring the life safety of the driver and passengers, and the present application has good system compatibility and can be adapted to more different car models.

[0091] Figure 2 FIG. 2 shows a structural block diagram of a driver status determination device 200 provided in an embodiment of the present application. Figure 2 As shown, the driver state determination device 200 includes:

[0092] The first determination module 210 is used to determine the target micro-expression type of the driver in the vehicle at the current moment based on the collected target facial image data of the driver in the vehicle and the trained micro-expression recognition model, wherein the target facial image data is used to represent the preprocessed facial image data.

[0093] The second determination module 220 is used to perform speech processing on the collected target speech stream data of the driver in the car based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the car at the current moment, wherein the labeled speech stream data is used to represent the preprocessed speech stream data.

[0094] The third determination module 230 is configured to determine the target emotion type of the driver at the current moment based on the target micro-expression type, target semantics, and target emotion type.

[0095] The fourth determining module 240 is configured to determine the target psychological state of the driver at the current moment based on the target emotion type and a preset emotion psychological state mapping table.

[0096] The fifth determining module 250 is configured to determine driving guidance information for providing psychological guidance to the driver in the vehicle based on the target psychological state.

[0097] Exemplarily, the second determining module 220 is specifically configured to:

[0098] The collected target voice stream data of the driver in the car is input into the trained voice recognition model for voice recognition to determine the text data corresponding to the target voice stream data.

[0099] Based on text data and preset natural language processing algorithms, semantic analysis is performed on the text data to determine the target semantics of the driver in the car at the current moment.

[0100] Based on text data and the trained sentiment classification model, the target emotion type of the driver in the car at the current moment is determined.

[0101] Exemplarily, the third determining module 230 is specifically configured to:

[0102] Based on the trained preset multi-layer perception neural network model, feature fusion and emotion classification are performed on the target micro-expression type, target semantics, and target emotion type to determine the target emotion type of the driver in the car at the current moment.

[0103] Exemplarily, the fifth determining module 250 is specifically configured to:

[0104] Based on the target psychological state and the preset risk level assessment model, determine the target risk level corresponding to the target psychological state.

[0105] The target psychological state and target risk level are input into a preset large language model to determine driving guidance information for providing psychological prompts and guidance to the driver in the car.

[0106] Exemplarily, the target psychological state and target risk level are input into a preset large language model to determine driving guidance information for providing psychological guidance to the driver in the vehicle, including:

[0107] If the target risk level is greater than the preset risk level, the target psychological state and the target risk level are input into the preset large language model, and driving guidance information is output, where the driving guidance information includes psychological guidance information for psychological guidance of the driver in the vehicle and auxiliary driving information for auxiliary prompts to the driver in the vehicle; or

[0108] If the target risk level is less than or equal to the preset risk level, the target psychological state and the target risk level are input into the preset large language model, and driving guidance information is output, which includes psychological guidance information.

[0109] Exemplarily, the driver state determining device 200 is further configured to:

[0110] Obtain the driver's feedback behavior in the car regarding psychological counseling information and / or assisted driving information.

[0111] Based on feedback behavior, the preset large language model is optimized.

[0112] Exemplarily, the driver state determining device 200 is further configured to:

[0113] Acquire initial facial image data and initial voice stream data of the driver in the car.

[0114] Grayscale processing and noise reduction processing are performed on the initial facial image data to determine the target facial image data of the driver in the car.

[0115] The initial voice stream data is subjected to noise reduction and dereverberation processing to determine the target voice stream data of the driver in the car.

[0116] The driver state determination device 200 provided in the embodiment of the present application, compared with the prior art, determines the target micro-expression type of the driver in the car at the current moment based on the collected target facial image data of the driver in the car and the trained micro-expression recognition model, and then performs voice processing on the collected target voice stream data of the driver in the car based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the car at the current moment, and then determines the target emotion type of the driver in the car at the current moment based on the target micro-expression type, target semantics and target emotion type, and then determines the target psychological state of the driver in the car at the current moment based on the target emotion type and a preset emotion-psychological state mapping table, and finally determines driving guidance information for psychological prompts and guidance for the driver in the car based on the target psychological state. The embodiment provided by the present application can conduct in-depth analysis and effective guidance and intervention on the psychological state of the driver in the car, thereby improving the accuracy of the monitoring results of the driver's state monitoring, further ensuring the life safety of the driver and passengers, and the present application has good system compatibility and can be adapted to more different car models.

[0117] See also Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the electronic device 300 includes a processor 310 , a memory 320 , and a bus 330 .

[0118] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 via the bus 330. When the machine-readable instructions are executed by the processor 310, the above-mentioned Figure 1 The specific implementation of the steps of the driver status determination method in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0119] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps of the driver status determination method in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0120] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0122] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-readable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] An embodiment of the present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of the driver status determination method.

[0127] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state drive (SSD)), etc.

[0128] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0130] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0134] Although the preferred embodiments of this specification have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this specification.

[0135] Obviously, those skilled in the art may make various changes and modifications to this specification without departing from the spirit and scope of this specification. Thus, if such changes and modifications fall within the scope of the claims of this specification and their equivalents, this specification is intended to include such changes and modifications.

Claims

1. A method for determining a driver's state, characterized in that: The method comprises: Determining a target micro-expression type of the driver in the vehicle at a current moment based on the collected target facial image data of the driver in the vehicle and the trained micro-expression recognition model; Based on a preset natural language processing algorithm, speech processing is performed on the collected target speech stream data of the driver in the vehicle to determine the target semantics and target emotion type of the driver in the vehicle at the current moment; Determining a target emotion type of the driver at the current moment based on the target micro-expression type, the target semantics, and the target emotion type; Determining the target psychological state of the driver at the current moment based on the target emotion type and a preset emotion psychological state mapping table; Based on the target psychological state, driving guidance information for providing psychological prompts and guidance to the driver in the vehicle is determined.

2. The driver status determination method according to claim 1, characterized in that: The performing speech processing on the collected target speech stream data of the driver in the vehicle based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the vehicle at the current moment includes: Inputting the collected target voice stream data of the driver in the vehicle into a trained voice recognition model for voice recognition, and determining text data corresponding to the target voice stream data; Performing semantic analysis on the text data based on the text data and a preset natural language processing algorithm to determine the target semantics of the driver in the vehicle at the current moment; Based on the text data and the trained emotion classification model, a target emotion type of the driver in the vehicle at the current moment is determined.

3. The driver status determination method according to claim 1, characterized in that: The determining, based on the target micro-expression type, the target semantics, and the target emotion type, the target emotion type of the driver at the current moment includes: Based on a trained preset multi-layer perceptual neural network model, feature fusion and emotion classification are performed on the target micro-expression type, the target semantics, and the target emotion type to determine the target emotion type of the driver in the car at the current moment.

4. The method for determining the driver's state according to claim 1, wherein: The determining, based on the target psychological state, driving guidance information for providing psychological guidance to the driver in the vehicle includes: Determining a target risk level corresponding to the target psychological state based on the target psychological state and a preset risk level assessment model; The target psychological state and the target risk level are input into a preset large language model to determine driving guidance information for providing psychological prompts and guidance to the driver in the vehicle.

5. The driver status determination method according to claim 4, characterized in that: The step of inputting the target psychological state and the target risk level into a preset large language model to determine driving guidance information for providing psychological guidance to the driver in the vehicle includes: If the target risk level is greater than a preset risk level, the target psychological state and the target risk level are input into a preset large language model, and the driving counseling information is output, where the driving counseling information includes psychological counseling information for psychological counseling of the driver in the vehicle and auxiliary driving information for providing auxiliary prompts to the driver in the vehicle; or If the target risk level is less than or equal to the preset risk level, the target psychological state and the target risk level are input into a preset large language model, and the driving guidance information is output, where the driving guidance information includes the psychological guidance information.

6. The method for determining the driver's state according to claim 5, characterized in that: After inputting the target psychological state and the target risk level into a preset large language model to determine driving guidance information for providing psychological guidance to the driver in the vehicle, the determining method further includes: Obtaining feedback behavior of the in-vehicle driver in response to the psychological counseling information and / or the assisted driving information; Based on the feedback behavior, the preset large language model is optimized.

7. The driver status determination method according to claim 1, characterized in that: Before determining the target micro-expression type of the driver at the current moment based on the collected target facial image data of the driver in the vehicle and the trained micro-expression recognition model, the determination method further includes: Acquire initial facial image data and initial voice stream data of the driver in the car; Performing grayscale processing and noise reduction processing on the initial facial image data to determine target facial image data of the driver in the vehicle; Noise reduction and dereverberation processing are performed on the initial voice stream data to determine target voice stream data of the driver in the vehicle.

8. A driver status determination device, characterized in that: The driver state determining device comprises: a first determination module, configured to determine a target micro-expression type of the driver in the vehicle at a current moment based on collected target facial image data of the driver in the vehicle and a trained micro-expression recognition model, wherein the target facial image data is used to represent the preprocessed facial image data; a second determination module, configured to perform speech processing on the collected target speech stream data of the driver in the vehicle based on a preset natural language processing algorithm to determine the target semantics and target emotion type of the driver in the vehicle at the current moment, wherein the target speech stream data is used to represent the preprocessed speech stream data; A third determination module is configured to determine a target emotion type of the driver at the current moment based on the target micro-expression type, the target semantics, and the target emotion type; A fourth determining module is configured to determine the target psychological state of the driver at the current moment based on the target emotion type and a preset emotion psychological state mapping table; The fifth determining module is used to determine driving guidance information for providing psychological prompts and guidance to the driver in the vehicle based on the target psychological state.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to perform the steps of the driver status determination method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the driver status determination method according to any one of claims 1 to 7.

Images