Driver emotion prediction method and device and computer readable storage medium

Abstract

The invention relates to a driver emotion prediction method and device and a computer readable storage medium, and the method comprises the steps: generating multi-type data according to collected original data; the original data comprises expression information of a driver, behavior information of the driver and vehicle information; the multi-type data comprises time and space data, expression behavior data and psychological state data; generating a behavior flow chart according to the multi-type data; calculating the similarity between the sub-graph of the behavior flow chart and the sub-graph of a pre-generated emotion behavior graph; and generating the emotion of the driver according to a similarity calculation result. Through the method and the device, the problem of inaccurate emotion prediction of the driver in related technologies is solved, and accurate prediction of the emotion of the driver is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a driver emotion prediction method, device and computer-readable storage medium. Background Art

[0002] At present, driver emotion recognition technology mainly focuses on facial expression recognition and physiological signal analysis. In terms of facial expression recognition, traditional methods mostly use static analysis based on geometric features or appearance features. Although they have achieved certain results in laboratory environments, they are often difficult to cope with interference from complex factors such as light changes and head movements in actual driving scenarios. While physiological signal analysis methods, such as heart rate variability analysis and galvanic skin response detection, can provide more objective emotional indicators, they often require the wearing of special sensor equipment, which causes inconvenience to drivers and is difficult to be widely used in daily driving.

[0003] In addition, most existing emotion recognition systems use a single data source, which makes it difficult to fully capture the complexity and dynamic changes of the driver's emotional state. For example, some systems rely only on facial expressions to judge emotions, ignoring important information such as driving behavior, making the driver's emotion prediction inaccurate.

[0004] Currently, no effective solution has been proposed to the problem of inaccurate driver emotion prediction in related technologies. Summary of the invention

[0005] In this embodiment, a driver emotion prediction method, device, and computer-readable storage medium are provided to solve the problem of inaccurate driver emotion prediction in the related art.

[0006] In a first aspect, a driver emotion prediction method is provided in this embodiment, the method comprising:

[0007] Generate multi-type data based on the collected raw data; the raw data includes the driver's expression information, the driver's behavior information and vehicle information; the multi-type data includes time and space data, expression behavior data and psychological state data;

[0008] generating a behavior flow chart according to the multi-type data;

[0009] Calculating the similarity between the subgraph of the behavior flow graph and the subgraph of the pre-generated emotion behavior graph;

[0010] The driver's emotion is generated according to the similarity calculation result.

[0011] In some embodiments, generating a behavior flow chart according to the multi-type data includes:

[0012] generating the behavior data according to the raw data;

[0013] The behavior flow chart is generated according to the occurred behavior data and the multi-type data.

[0014] In some embodiments, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart includes:

[0015] Extracting a first feature vector from each time node of the subgraph of the behavior flowchart; the first feature vector is used to characterize the degree distribution, node type distribution and edge type distribution of the nodes of the subgraph of the behavior flowchart;

[0016] Normalizing the first feature vector to obtain a first probability distribution;

[0017] Extracting a second feature vector from each time node of the subgraph of the emotional behavior graph; the second feature vector is used to characterize the degree distribution, node type distribution and edge type distribution of the nodes of the subgraph of the emotional behavior graph;

[0018] Normalizing the second eigenvector to obtain a second probability distribution;

[0019] The similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart is calculated according to the first probability distribution and the second probability distribution.

[0020] In some embodiments, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart according to the first probability distribution and the second probability distribution includes:

[0021] Calculating the KL divergence of the first probability distribution to obtain a first KL divergence;

[0022] Calculate the KL divergence of the second probability distribution to obtain a second KL divergence;

[0023] The similarity between the subgraph of the behavior flow graph and the subgraph of the emotion behavior graph is calculated according to the first KL divergence and the second KL divergence.

[0024] In some embodiments, calculating the KL divergence of the first probability distribution to obtain the first KL divergence includes: obtaining the first KL divergence by integrating the first probability distribution in time and space;

[0025] Calculating the KL divergence of the second probability distribution to obtain the second KL divergence includes: obtaining the second KL divergence by integrating the second probability distribution in time and space.

[0026] In some embodiments, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart includes:

[0027] Extracting a third feature vector from each time node of the subgraph of the behavior flowchart; the third feature vector is used to characterize the degree distribution, node type distribution, edge type distribution and meta-path of the nodes of the subgraph of the behavior flowchart;

[0028] Normalizing the third eigenvector to obtain a third probability distribution;

[0029] Extracting a fourth eigenvector from each time node of the subgraph of the emotional behavior graph; the fourth eigenvector is used to characterize the degree distribution, node type distribution, edge type distribution and meta-path of the nodes of the subgraph of the emotional behavior graph;

[0030] Normalizing the fourth eigenvector to obtain a fourth probability distribution;

[0031] The similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart is calculated according to the first probability distribution, the second probability distribution, the third probability distribution and the fourth probability distribution.

[0032] In some embodiments, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart according to the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution includes:

[0033] Calculating the KL divergence of the third probability distribution to obtain a third KL divergence;

[0034] Calculating the KL divergence of the fourth probability distribution to obtain a fourth KL divergence;

[0035] The similarity between the subgraph of the behavior flow graph and the subgraph of the emotion behavior graph is calculated according to the first KL divergence, the second KL divergence, the third KL divergence and the fourth KL divergence.

[0036] In some embodiments, generating the driver's emotion according to the similarity calculation result includes:

[0037] According to the similarity calculation result, determining the subgraph of the emotion behavior graph having the highest similarity to the subgraph of the behavior flow graph;

[0038] The driver's emotion is determined according to the emotion information of the subgraph of the emotion behavior graph having the highest similarity to the subgraph of the behavior flow graph.

[0039] In a second aspect, a driver emotion prediction device is provided in this embodiment, the device comprising:

[0040] A collection module, used to generate multiple types of data according to the collected raw data; the raw data includes the driver's expression information, the driver's behavior information and vehicle information; the multiple types of data include the time and space data, expression behavior data and psychological state data;

[0041] A first generating module, used for generating a behavior flow chart according to the multi-type data;

[0042] A calculation module, used for calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart;

[0043] The second generating module is used to generate the driver's emotion according to the similarity calculation result.

[0044] In a third aspect, in this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the driver emotion prediction method described in the first aspect are performed.

[0045] Compared with the related art, the driver emotion prediction method, device and computer-readable storage medium provided in this embodiment generate multiple types of data based on the collected original data; the original data includes the driver's expression information, the driver's behavior information and the vehicle information; the multiple types of data include time and space data, expression behavior data and psychological state data; a behavior flow chart is generated based on the multiple types of data; the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart is calculated; the driver's emotions are generated based on the similarity calculation result, which solves the problem of inaccurate driver emotion prediction existing in the related art and realizes accurate prediction of the driver's emotions.

[0046] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 is a hardware structure block diagram of a terminal of a driver emotion prediction method provided in this embodiment;

[0049] Figure 2 is a flow chart of a driver emotion prediction method provided by an embodiment of the present application;

[0050] Figure 3A It is a schematic diagram of a behavior flow chart provided in an embodiment of the present application;

[0051] Figure 3B is a schematic diagram of another behavior flow chart provided in an embodiment of the present application;

[0052] Figure 3C is a schematic diagram of another behavior flow chart provided in an embodiment of the present application;

[0053] Figure 3D is a schematic diagram of another behavior flow chart provided in an embodiment of the present application;

[0054] Figure 3E is a schematic diagram of another behavior flow chart provided in an embodiment of the present application;

[0055] Figure 3F is a schematic diagram of another behavior flow chart provided in an embodiment of the present application;

[0056] Figure 4A is a schematic diagram of an emotional behavior diagram provided in an embodiment of the present application;

[0057] Figure 4B is a schematic diagram of another emotional behavior diagram provided in an embodiment of the present application;

[0058] Figure 4C is a schematic diagram of another emotional behavior diagram provided in an embodiment of the present application;

[0059] Figure 4D is a schematic diagram of another emotional behavior diagram provided in an embodiment of the present application;

[0060] Figure 4E is a schematic diagram of another emotional behavior diagram provided in an embodiment of the present application;

[0061] Figure 4F is a schematic diagram of another emotional behavior diagram provided in an embodiment of the present application;

[0062] Figure 4G is a schematic diagram of another emotional behavior diagram provided in an embodiment of the present application;

[0063] Figure 4H is a schematic diagram of another emotional behavior diagram provided in an embodiment of the present application;

[0064] Figure 5 It is a structural block diagram of a driver emotion prediction device of this embodiment. DETAILED DESCRIPTION

[0065] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0066] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0067] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of a driver emotion prediction method provided in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0068] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a driver emotion prediction method in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0069] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0070] In this embodiment, a driver emotion prediction method is provided. Figure 2 is a flowchart of a driver emotion prediction method provided by an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0071] Step S210, generating multiple types of data based on the collected original data; the original data includes the driver's expression information, the driver's behavior information and the vehicle information; the multiple types of data include time and space data, expression behavior data and psychological state data.

[0072] In this step, the processor or controller generates multiple types of data based on the collected raw data. The raw data here include the driver's facial expression information, the driver's behavior information and the vehicle information. The raw data here are multimodal data, such as image data, text data, voice data, etc. For example, the driver's facial expression information includes eye information, eyebrow information, mouth information, chin information, nose information, cheek information, etc.; the driver's behavior information includes upper limb body movement information, etc.; the vehicle information includes the vehicle's driving state information, such as vehicle acceleration, vehicle uniform speed situation, vehicle emergency braking, vehicle steering and other vehicle state information. The multi-type data includes time and space data, facial expression behavior data and psychological state data. The spatial data here is used to characterize the moving state of the car, and the time data here is used to characterize the driver's facial expression information at different times or the vehicle information of the vehicle at different times. The time and space data here also include the behavior data that occurred, such as following the car too close, distracted driving, incorrect lane use, etc. The facial expression behavior data here include text information and image information representing the driver's facial expression, and the facial expression behavior data also include the degree of behavior of the driver's facial features, such as a large dilation of the pupil, a slight forward lean of the body, etc. The facial expression behavior data also include the driver's facial expression information and body movement information, etc. The mental state data here include text information and image information of the mental state, and the text information of the mental state here can be extracted from the original data, such as extracted from the image data representing the driver's facial expression information.

[0073] Step S220, generating a behavior flow chart according to the multi-type data.

[0074] In this step, the processor or controller generates a behavior flow chart based on multiple types of data, and the behavior flow chart includes the driver's facial expression behavior information and the behavior data that occurred. Figure 3A to Figure 3F ,in, Figure 3A , 3B , 3C is a behavior flow chart of simultaneous actions, Figure 3D , 3E 3F is a behavior flow chart of sequential actions. Further, generating a behavior flow chart based on multiple types of data includes: generating the behavior data that occurred based on the original data; and generating a behavior flow chart based on the behavior data that occurred and the multiple types of data.

[0075] Step S230, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart.

[0076] In this step, the processor or controller calculates the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart. The subgraph of the behavior flow chart here can be Figure 3A to Figure 3F Any behavior flow chart in Figure 3A to Figure 3FAny behavior in the flowchart includes a subgraph of some nodes, and can also be Figure 3D to Figure 3F A sub-image represented by one of the frames in , for example Figure 3E The sub-graph of the second frame before the speeding behavior. The pre-generated emotional behavior graph here can be established based on psychological knowledge, and the pre-generated emotional behavior graph is as follows: Figure 4A to Figure 4H As shown, the subgraph of the emotional behavior graph can be Figure 4A to Figure 4H Any emotional behavior graph can also be Figure 4A to Figure 4H Any emotional behavior graph includes a subgraph of some nodes. This embodiment focuses on the description and depiction of facial expressions and upper limb movements, matches the emotions of this graph with the recorded facial expressions and movements, and then predicts the driving behavior (at this time, it no longer focuses on what kind of driving behavior but directly judges the level of danger at 3 levels). The nodes are divided into (expression state-expression-danger level). The first edge relationship is the degree level (slightly / largely / severely); the second edge is the degree correspondence relationship. The detection technology is combined with the time series of facial expressions and movements (simultaneous and sequential movements) to assist in judging driving behavior.

[0077] Among them, calculating the similarity between a subgraph of a behavior flow graph and a subgraph of a pre-generated emotion behavior graph includes: extracting a first eigenvector from each time node of the subgraph of the behavior flow graph; the first eigenvector is used to characterize the degree distribution, node type distribution and edge type distribution of the nodes of the subgraph of the behavior flow graph; normalizing the first eigenvector to obtain a first probability distribution; extracting a second eigenvector from each time node of the subgraph of the emotion behavior graph; the second eigenvector is used to characterize the degree distribution, node type distribution and edge type distribution of the nodes of the subgraph of the emotion behavior graph; normalizing the second eigenvector to obtain a second probability distribution; and calculating the similarity between the subgraph of the behavior flow graph and the subgraph of the emotion behavior graph based on the first probability distribution and the second probability distribution.

[0078] Exemplarily, feature vectors are extracted from each time node of a heterogeneous graph, where the heterogeneous graphs are subgraphs of a behavior flow graph and subgraphs of an emotion behavior graph. These features may include node degree distribution, node type distribution, edge type distribution, etc. For example, for node type distribution, the proportion of different types of nodes under each time / space node of the two subgraphs may be calculated to form a probability distribution vector. The probability distribution vector may be formed by calculating the proportion of different types of nodes such as body parts, actions, and psychological states, or the degree distribution of nodes. Feature vectors may also be constructed by selecting multiple features of nodes and edges. For example, feature vectors [p1, p2, p3, q1, q2], where p1, p2, and p3 represent body parts, psychological states, and proportions of physical performance, q1 represents the degree, and q2 represents the occurrence of emotions / behaviors. According to the form of the feature vector, a first feature vector and a second feature vector are constructed, that is, the first feature vector and the second feature vector both include features such as body parts, psychological states, proportions of physical performance, degree, and occurrence of emotions / behaviors. The extracted feature vector is converted into a probability distribution, that is, the first feature vector and the second feature vector are converted into a first probability distribution and a second probability distribution, respectively. If the feature vector itself is not a probability distribution, it can be converted into a probability distribution through normalization or other methods. For example, for the node degree distribution, the proportion of the number of nodes corresponding to each degree value to the total number of nodes can be calculated to form a probability distribution.

[0079] Wherein, according to the first probability distribution and the second probability distribution, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart, includes: calculating the KL divergence of the first probability distribution to obtain the first KL divergence; calculating the KL divergence of the second probability distribution to obtain the second KL divergence; according to the first KL divergence and the second KL divergence, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart. Further, calculating the KL divergence of the first probability distribution to obtain the first KL divergence includes: integrating the first probability distribution in time and space to obtain the first KL divergence; calculating the KL divergence of the second probability distribution to obtain the second KL divergence includes: integrating the second probability distribution in time and space to obtain the second KL divergence. KL divergence (Kullback-Leibler divergence) can be called relative entropy or information divergence. The theoretical significance of KL divergence is to measure the difference between two probability distributions. When the KL divergence is larger, the difference between the two is larger; when the KL divergence is smaller, the difference between the two is smaller. If the two are the same, the KL divergence should be 0.

[0080] Construct a probability density function. In order to reflect the continuity of spatiotemporal attributes, you can use Gaussian processes or other spatiotemporal models to construct a probability density function. For example: Gaussian process: Assuming that it obeys a Gaussian process, its mean and covariance matrix can change over time and space. Spatiotemporal mixed model: Combine time series models (such as ARIMA) and spatial models (such as geographically weighted regression) to define probability density. Substitute the constructed probability density function into the integral formula of the KL divergence, and calculate its value by numerical integration methods (such as Monte Carlo integration). Exemplarily, the first KL divergence D is calculated according to the following formula KL (P||Q) and the second KL divergence D KL (Q||P).

[0081]

[0082] In the above formula, P(x(t,s)) and Q(x(t,s)) are two probability density functions, that is, P(x(t,s)) and Q(x(t,s)) are the first probability distribution and the second probability distribution, respectively. P(x(t,s)) and Q(x(t,s)) can represent the probability distribution of two time nodes, respectively, and can reflect the similarity (continuity) of the behavior development, psychological state or emotion generation of the emotion behavior graph or behavior flow chart at different time nodes, thereby reflecting the immediacy and directly predicting emotions through psychological state and behavior performance. P(x(t,s)) and Q(x(t,s)) can also reflect the similarity of the two graph subgraphs at the same time node, reflecting the simultaneity, and predicting the emotions in the emotion behavior graph through the behavior flow chart.

[0083] Convert KL divergence to similarity, assuming that the two distributions are as follows: the first probability distribution P of the subgraph G1 of the behavior flow graph is (0.3, 0.2, 0.2, 0.28, 0.02), and the second probability distribution Q of the subgraph G2 of the emotional behavior graph is (0.2, 0.5, 0.15, 0.04, 0.11). First, calculate the first KL divergence D KL (P||Q) and the second KL divergence D KL (Q||P), and then add them together to get the symmetric KL divergence D sym (P||Q), and finally take the negative value to get the first similarity S sym1 (P,Q), as shown in the following formula.

[0084] D sym (P||Q)=D KL (P||Q)+D KL (Q||P)

[0085] S sym1 (P,Q)=-D sym (P||Q)

[0086] When the two degree distributions are exactly the same, S sym1 (P,Q) is 0. When the difference between the two degree distributions is larger, S sym1 The value of (P,Q) will be smaller. A high similarity indicates that the occurrence of the behavior (identified by existing recognition technology) and the emotional process caused by the behavior are similar, so as to achieve the purpose of inferring the driver's emotions from the behavior, so as to predict the subsequent dangerous driving behavior. For example, if the eyes are squinting sharply, the lips are sharply turned down, and the nose is sharply expanded, it is predicted that the driver has an intolerable psychological state, and the emotions of hatred and rage can be predicted. After weighted processing of all data, it is predicted that such emotions will lead to corresponding dangerous driving behaviors.

[0087] Among them, calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart includes: extracting a third eigenvector from each time node of the subgraph of the behavior flow chart; the third eigenvector is used to characterize the degree distribution, node type distribution, edge type distribution and meta-path of the nodes of the subgraph of the behavior flow chart; normalizing the third eigenvector to obtain a third probability distribution; extracting a fourth eigenvector from each time node of the subgraph of the emotion behavior chart; the fourth eigenvector is used to characterize the degree distribution, node type distribution, edge type distribution and meta-path of the nodes of the subgraph of the emotion behavior chart; normalizing the fourth eigenvector to obtain a fourth probability distribution; and calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart based on the first probability distribution, the second probability distribution, the third probability distribution and the fourth probability distribution. Furthermore, based on the first probability distribution, the second probability distribution, the third probability distribution and the fourth probability distribution, the similarity between the subgraph of the behavior flowchart and the subgraph of the emotion behavior graph is calculated, including: calculating the KL divergence of the third probability distribution to obtain the third KL divergence; calculating the KL divergence of the fourth probability distribution to obtain the fourth KL divergence; calculating the similarity between the subgraph of the behavior flowchart and the subgraph of the emotion behavior graph based on the first KL divergence, the second KL divergence, the third KL divergence and the fourth KL divergence.

[0088] Further, a first similarity is calculated according to the first KL divergence and the second KL divergence, a second similarity is calculated according to the third KL divergence and the fourth KL divergence, and a similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart is calculated according to the first similarity and the second similarity.

[0089] Exemplarily, a meta-path is constructed, with node types and edge types, nodes: body parts, action states, psychological states, emotions, probability; edges: simultaneously / even if, leading to, inhibiting, reinforcing, degree adverbs / / defining the degree of the edge. Typical meta-path examples: body parts → (embody) performance → (lead to) driving behavior; description: body parts affect driving behavior through performance; case: nervous performance of eyes → (affect) unstable steering wheel control. Body parts → (embody) psychological state → (affect) driving behavior; description: body parts reflect psychological state, which in turn affects driving behavior; case: frowning of eyebrows → anxious state → impatient driving. Body parts → (embody) performance → (reflect) psychological state → (inhibit / reinforce) driving behavior; description: composite path, reflecting multi-dimensional association; case: trembling of mouth → nervous performance → panic state → overcorrection. Example extraction method: Random Walk strategy: a) Starting point selection: each node is used as the starting point; b) Walk constraints (rules used when traversing): strictly follow the meta-path node type conversion; ensure the semantic consistency of the path; control the walk length and depth; c) Walk algorithm pseudocode: i. For each starting node v; ii. According to the predefined meta-path P; iii. Randomly select neighbor nodes that meet the P type conversion; iv. Generate a node sequence of length k; v. Repeat multiple times to obtain a rich set of node sequences.

[0090] Extract node and edge features, node feature extraction: convert nodes in heterogeneous graphs into feature values. Heterogeneous graphs contain multiple types of nodes and edges, so it is necessary to extract features based on node types and contexts. Common methods include: Meta-path-based feature extraction: Generate node context information through meta-paths. For example, the meta-path of body part → (manifestation) performance → (reflection) psychological state → (inhibition / reinforcement) driving behavior, the upper and lower node information of the above four nodes can be produced through this path. For each subgraph, extract edge features based on the constructed meta-path. These features may include: Edge type distribution: calculate the proportion of each type of edge in the subgraph. Meta-path features: calculate the frequency or number of paths of each meta-path in the subgraph. Convert the extracted features into feature vectors. For example, assuming that the subgraph has 2 edge types, 2 node types, and 2 meta-paths, the feature vector can be expressed as: (p1, p2, q1, q2, h1, h2). Among them, p1, p2 respectively represent the proportion of the two edge types (such as simultaneous, leading to, reinforcing), q1, q2 respectively represent two types of nodes, and h1, h2 respectively represent the frequencies of the two meta-paths. According to the form of the eigenvector, the third eigenvector and the fourth eigenvector are constructed, that is, the third eigenvector and the fourth eigenvector both include the above features, p1, p2, q1, q2, h1, h2. The eigenvectors are normalized to probability distributions, that is, the third eigenvector and the fourth eigenvector are normalized to obtain the third probability distribution and the fourth probability distribution. The normalization method is to divide each eigenvalue by the sum of the eigenvectors so that the sum is 1.

[0091] Construct a probability density function. In order to reflect the continuity of spatiotemporal attributes, you can use Gaussian processes or other spatiotemporal models to construct a probability density function. For example: Gaussian process: Assuming that it obeys a Gaussian process, its mean and covariance matrix can change over time and space. Spatiotemporal mixed model: Combine time series models (such as ARIMA) and spatial models (such as geographically weighted regression) to define probability density. Substitute the constructed probability density function into the integral formula of the KL divergence and calculate its value through numerical integration methods (such as Monte Carlo integration). Calculate the third KL divergence D according to the following formula KL3 (P1||Q1) and the fourth KL divergence D KL4 (Q1||P1).

[0092]

[0093] In the above formula, P1(x(t,s)) and Q1(x(t,s)) are two probability density functions, that is, P1(x(t,s)) and Q1(x(t,s)) are the third probability distribution and the fourth probability distribution, respectively, and dt and ds represent the integrals of the time dimension and the space dimension, respectively. P1(x(t,s)) and Q1(x(t,s)) represent the node / edge type and meta-path probability distribution characteristics of the two subgraphs, respectively. When the similarity of node type, edge type or meta-path structure between the two graphs is high (similar node type proportion / similar edge type proportion / meta-path is highly consistent), such as similar proportions of reinforcement, inhibition, degree adverb types, etc. (all have large eye squints, exasperation, impatient reinforcement, etc., and similar meta-structures), it can be judged that P1 and Q1 have a high KL divergence.

[0094] Convert KL divergence to similarity, assuming that the two distributions are as follows: the first probability distribution P of the subgraph G1 of the behavior flow chart is (0.3, 0.2, 0.2, 0.28, 0.02), and the second probability distribution Q of the subgraph G2 of the emotional behavior chart is (0.2, 0.5, 0.15, 0.04, 0.11). First, calculate the third KL divergence D KL3 (P1||Q1) and the fourth KL divergence D KL4 (Q1||P1), and then add them together to get the symmetric KL divergence D sym1 (P||Q), and finally take the negative value to get the second similarity S sym2 (P,Q), as shown in the following formula.

[0095] D sym1 (P1||Q1)=D KL3 (P1||Q1)+D KL4 (Q1||P1)

[0096] S sym2 (P1,Q1)=-D sym1 (P1||Q1)

[0097] When the two degree distributions are exactly the same, S sym2 (P,Q) is 0. When the difference between the two degree distributions is larger, S sym2 The value of (P,Q) will be smaller. A high similarity indicates that the occurrence of the behavior (identified by existing recognition technology) and the emotional process caused by the behavior are similar, so as to achieve the purpose of inferring the driver's emotions from the behavior, so as to predict the subsequent dangerous driving behavior. For example, the eyes are squinting sharply, the lips are slanting sharply, the nose is expanding sharply, the intolerable psychological state is strengthened, and the eyes are looking straight ahead, etc., which can predict that the driver has hatred and rage. After weighted processing of all data, it is predicted that such emotions will lead to corresponding dangerous driving behaviors.

[0098] The first similarity and the second similarity are weighted to obtain the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotional behavior chart. The final similarity can be calculated by time weighting and feature weighting. Time weighting is to assign weights to samples in time series data to reflect their temporal importance. Application scenario: Time series prediction: Recent data may be more important than historical data. Data decay: The weight of old data can be gradually reduced. Implementation method: Exponential decay: Assign a weight that decays over time to each sample. Sliding window: Only assign non-zero weights to data in the most recent period. Feature weighting is to assign weights to each feature in the data set to reflect its importance to the model. Application scenario: Feature selection: Highlight important features and suppress unimportant features through weighting. Domain knowledge: Assign weights to features based on business knowledge. Implementation method: Manually set weights: Assign weights to features based on domain knowledge. Automatically calculate weights and use statistical methods (such as chi-square test, mutual information) to calculate feature importance. Use feature importance scores of models (such as linear regression, decision trees).

[0099] Step S240: generating the driver's emotion according to the similarity calculation result.

[0100] In this step, the processor or controller generates the driver's emotion according to the similarity calculation result. According to the similarity calculation result, the subgraph of the emotion behavior graph with the highest similarity to the subgraph of the behavior flow graph is determined, and the driver's emotion is determined according to the emotion information of the subgraph of the emotion behavior graph with the highest similarity to the subgraph of the behavior flow graph. The emotion information here is as follows: Figure 4A to Figure 4H The satisfaction / happiness / ecstasy, strangeness / surprise / amazement, etc.

[0101] Through the above steps, the spatiotemporal stream, expression behavior stream, psychological state stream and data stream are integrated in the form of integrals to embody the continuity of spatiotemporal attributes with the KL divergence calculation formula, and several data streams are integrated to jointly predict emotions. Specifically, emotions are directly predicted through expressions, psychological states and behavioral performances, driving behaviors are predicted through emotions, and the time series (simultaneous and sequential actions) of detecting expression actions are combined to assist in judging driving behaviors. Heterogeneous graphs are built using the emotion behavior graph and behavior flow chart established with psychological knowledge, and the actual collected information such as the driver's facial expression information and body movement information are used as subgraphs of the behavior flow chart. The subgraph of the emotion behavior graph is used as a paradigm for predicting emotions. By finding the similarity between the two subgraphs, emotions are predicted using expressions, actions, psychological states and driving behaviors. Data processing calculates the KL divergence and converts it into similarity. The similarities calculated in the two ways are weighted to obtain the final similarity. A high similarity indicates that the emotion behavior graph and the behavior flow chart subgraph are highly similar, and the probability of the emotion occurring is high. This solves the problem of inaccurate driver emotion prediction in the related art.

[0102] At the data collection level, this embodiment not only considers facial expressions, but also integrates multi-dimensional information of body movements and psychological states, and makes modifications based on the Plutchek three-dimensional model to refine emotions into different types and levels, greatly improving the comprehensiveness and accuracy of emotion recognition. Secondly, in terms of data analysis, heterogeneous graphs are introduced to build emotional behavior graphs and behavioral flow charts. The nonlinear relationship and dynamic changes between expressions, action psychological states and emotional characteristics can be effectively captured through subgraph similarity, overcoming the limitations of traditional linear analysis methods. More importantly, this embodiment integrates the spatiotemporal sequence flow, expression behavior flow, psychological state flow and data flow, and integrates the KL divergence calculation formula in the form of integrals to reflect the continuity of spatiotemporal attributes; several data streams are integrated to jointly predict emotions. This system can not only identify the current emotional state, but also predict future emotional change trends, providing the possibility of timely intervention in potential risks.

[0103] In addition, based on the facial expression information and body movement information collected from the driver, the closest one is matched from the 8 psychological state diagrams, and then the driver's driving behavior is predicted based on the matched psychological state diagram. For example, rage can be used to infer that chasing driving and vehicle out of control may occur; anger can be used to infer that behaviors such as not giving way to pedestrians, sudden braking, swerving the steering wheel, and incorrect lane use may occur; annoyance can be used to infer that speeding, low-speed driving, and following vehicles too closely may occur; boredom can be used to infer that behaviors such as not giving way to pedestrians, chasing, violating traffic lights, and incorrect lane use may occur; hatred can be used to infer that behaviors such as sudden braking, chasing, and vehicle out of control may occur; superiority can be used to infer that behaviors such as low-speed driving and incorrect lane use may occur; and fear can be used to infer that behaviors such as fear and fear may occur. Based on the prediction, the person may suddenly step on the brake pedal; based on fear, the person may swerve the steering wheel or lose control of the vehicle; based on depression, the person may use the lane incorrectly, exceed the speed limit, or drive at a low speed; based on surprise, the person may violate traffic lights, suddenly step on the brake pedal, swerve the steering wheel, use the lane incorrectly, or lose control of the vehicle; based on acceptance, the person may suddenly step on the brake pedal, drive at a low speed, or follow the vehicle too closely; based on ecstasy, the person may violate traffic lights, exceed the speed limit, chase the vehicle too close, lose control of the vehicle, swerve the steering wheel, etc.

[0104] In this embodiment, a driver emotion prediction system is also provided, including an information collection terminal for collecting the driver's facial expression information, body movement information and certain psychological state information; a psychological and emotional state analysis subsystem for receiving and analyzing the information transmitted by the information collection terminal, and judging the driver's emotions and the process of emotional changes; constructing a heterogeneous graph with the above-mentioned emotional behavior graph and behavioral flow chart established with psychological knowledge, and using the driver's facial expression information and body movement information actually collected as a sub-graph of the sub-graph of the behavioral flow chart, and the sub-graph of the emotional behavior graph as a paradigm for predicting emotions, by finding the similarity between the two sub-graphs, using expressions, movements, psychological states, and driving behaviors to predict emotions; and an early warning subsystem for issuing early warnings based on the analysis results of the psychological and emotional state analysis system platform.

[0105] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0106] In this embodiment, a driver emotion prediction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0107] Figure 5 is a structural block diagram of a driver emotion prediction device of this embodiment, such as Figure 5 As shown, the device comprises:

[0108] The acquisition module 510 is used to generate multiple types of data based on the collected raw data; the raw data includes the driver's expression information, the driver's behavior information and the vehicle information; the multiple types of data include time and space data, expression behavior data and psychological state data;

[0109] A first generating module 520, for generating a behavior flow chart according to multiple types of data;

[0110] A calculation module 530, used to calculate the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart;

[0111] The second generating module 540 is used to generate the driver's emotion according to the similarity calculation result.

[0112] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0113] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0114] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0115] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0116] S1, generating multiple types of data based on the collected raw data; the raw data includes the driver's expression information, the driver's behavior information and the vehicle information; the multiple types of data include time and space data, expression behavior data and psychological state data;

[0117] S2, generating a behavior flow chart based on multi-type data;

[0118] S3, calculating the similarity between the subgraph of the behavior flow graph and the subgraph of the pre-generated emotion behavior graph;

[0119] S4, generating the driver's emotion according to the similarity calculation result.

[0120] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0121] In addition, in combination with a driver emotion prediction method provided in the above embodiment, a storage medium can also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any driver emotion prediction method in the above embodiment is implemented.

[0122] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0123] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0124] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0125] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A driver emotion prediction method, characterized in that: The method comprises: Generate multi-type data based on the collected raw data; the raw data includes the driver's expression information, the driver's behavior information and vehicle information; the multi-type data includes time and space data, expression behavior data and psychological state data; generating a behavior flow chart according to the multi-type data; Calculating the similarity between the subgraph of the behavior flow graph and the subgraph of the pre-generated emotion behavior graph; The driver's emotion is generated according to the similarity calculation result.

2. The driver emotion prediction method according to claim 1, characterized in that: The generating a behavior flow chart according to the multi-type data includes: generating the behavior data according to the raw data; The behavior flow chart is generated according to the occurred behavior data and the multi-type data.

3. The driver emotion prediction method according to claim 1, characterized in that: The calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart comprises: Extracting a first feature vector from each time node of the subgraph of the behavior flowchart; the first feature vector is used to characterize the degree distribution, node type distribution and edge type distribution of the nodes of the subgraph of the behavior flowchart; Normalizing the first feature vector to obtain a first probability distribution; Extracting a second feature vector from each time node of the subgraph of the emotional behavior graph; the second feature vector is used to characterize the degree distribution, node type distribution and edge type distribution of the nodes of the subgraph of the emotional behavior graph; Normalizing the second eigenvector to obtain a second probability distribution; The similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart is calculated according to the first probability distribution and the second probability distribution.

4. The driver emotion prediction method according to claim 3, characterized in that: The calculating, according to the first probability distribution and the second probability distribution, the similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart comprises: Calculating the KL divergence of the first probability distribution to obtain a first KL divergence; Calculate the KL divergence of the second probability distribution to obtain a second KL divergence; The similarity between the subgraph of the behavior flow graph and the subgraph of the emotion behavior graph is calculated according to the first KL divergence and the second KL divergence.

5. The driver emotion prediction method according to claim 4, characterized in that: The calculating the KL divergence of the first probability distribution to obtain the first KL divergence includes: obtaining the first KL divergence by integrating the first probability distribution in time and space; Calculating the KL divergence of the second probability distribution to obtain the second KL divergence includes: obtaining the second KL divergence by integrating the second probability distribution in time and space.

6. The driver emotion prediction method according to claim 4 or 5, characterized in that: The calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart comprises: Extracting a third feature vector from each time node of the subgraph of the behavior flowchart; the third feature vector is used to characterize the degree distribution, node type distribution, edge type distribution and meta-path of the nodes of the subgraph of the behavior flowchart; Normalizing the third eigenvector to obtain a third probability distribution; Extracting a fourth eigenvector from each time node of the subgraph of the emotional behavior graph; the fourth eigenvector is used to characterize the degree distribution, node type distribution, edge type distribution and meta-path of the nodes of the subgraph of the emotional behavior graph; Normalizing the fourth eigenvector to obtain a fourth probability distribution; The similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart is calculated according to the first probability distribution, the second probability distribution, the third probability distribution and the fourth probability distribution.

7. The driver emotion prediction method according to claim 6, characterized in that: The calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the emotion behavior chart according to the first probability distribution, the second probability distribution, the third probability distribution and the fourth probability distribution includes: Calculating the KL divergence of the third probability distribution to obtain a third KL divergence; Calculating the KL divergence of the fourth probability distribution to obtain a fourth KL divergence; The similarity between the subgraph of the behavior flow graph and the subgraph of the emotion behavior graph is calculated according to the first KL divergence, the second KL divergence, the third KL divergence and the fourth KL divergence.

8. The driver emotion prediction method according to claim 1, characterized in that: Generating the driver's emotion according to the similarity calculation result includes: According to the similarity calculation result, determining the subgraph of the emotion behavior graph having the highest similarity to the subgraph of the behavior flow graph; The driver's emotion is determined according to the emotion information of the subgraph of the emotion behavior graph having the highest similarity to the subgraph of the behavior flow graph.

9. A driver emotion prediction device, characterized in that: The device comprises: A collection module, used to generate multiple types of data according to the collected raw data; the raw data includes the driver's expression information, the driver's behavior information and vehicle information; the multiple types of data include the time and space data, expression behavior data and psychological state data; A first generating module, used for generating a behavior flow chart according to the multi-type data; A calculation module, used for calculating the similarity between the subgraph of the behavior flow chart and the subgraph of the pre-generated emotion behavior chart; The second generating module is used to generate the driver's emotion according to the similarity calculation result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the driver emotion prediction method according to any one of claims 1 to 8 are implemented.

Images