A driver emotional state detection method and system based on upper body posture

Abstract

The present invention discloses a method and system for detecting the emotional state of a driver based on upper body posture, which relates to the field of computer vision technology, including: identifying key human body parts in a video clip of the upper body of a driver to be detected based on a preset human body posture estimation model, and obtaining a time series of each joint position; preprocessing the joint position time series, and converting it into joint flow data and bone flow data; inputting the joint flow and bone flow data into multiple trained graph convolutional neural network models, respectively, to obtain multiple classification results; fusing multiple classification results based on an integrated learning method, and obtaining the emotional state recognition result of the driver to be detected. The present invention alleviates the technical problem of poor accuracy of action emotion recognition in a driving environment in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for detecting a driver's emotional state based on upper body posture. Background Art

[0002] In modern society, traffic safety remains a high-profile issue. Extreme driver emotions have long been one of the main causes of road accidents worldwide. However, most current driver assistance systems ignore the impact of emotions on driving behavior and traffic safety. In the smart cockpit, accurate perception, understanding, and management of the driver's extreme emotions through various interactive strategies can effectively reduce the risk of accidents caused by such emotions, thereby improving road traffic safety.

[0003] Research on emotion recognition has mainly focused on sound, facial expressions, and EEG signals. However, human motion, one of the most basic and natural non-verbal ways of emotional communication, has not received due attention. Emotion recognition based on human motion has several advantages. First, compared with spoken language and text, human motion shows greater universality and cross-cultural consistency. Second, compared with facial expressions, human motion can better express a person's true emotions because motion is more difficult to suppress without training. Finally, compared with physiological signals (such as EEG signals), human motion provides a more direct and less intrusive method in the data collection process, preserving the natural behavior of the subjects.

[0004] In a semi-enclosed cockpit, the camera can only capture the upper body movements of the human body in most cases. At the same time, with limited onboard computing resources, the intelligent system requires a method with lower data complexity and higher computing efficiency, and can protect personal privacy to a certain extent. Data sets collected from real scenes often have unbalanced data distribution, that is, a few categories contain a large number of data samples.

[0005] Most existing methods rely on full-body poses and graph convolutional networks with predetermined adjacency matrices, facing challenges in driving scenarios, including limited visibility, constrained motion, and unbalanced data distribution, which affect the generalization ability and accuracy of the models. Summary of the invention

[0006] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention provides a method and system for detecting the emotional state of a driver based on upper body posture. The technical solution is as follows:

[0007] On the one hand, a method for detecting a driver's emotional state based on upper body posture is provided, the method comprising: identifying key human body parts in an upper body video clip of a driver to be detected based on a preset human body posture estimation model, and obtaining a time series of joint positions; preprocessing the joint position time series and converting them into joint flow data and skeleton flow data; inputting the joint flow data and the skeleton flow data into multiple trained graph convolutional neural network models, respectively, to obtain multiple classification results; and fusing the multiple classification results based on an ensemble learning method to obtain an emotional state recognition result of the driver to be monitored.

[0008] Furthermore, the graph convolutional neural network model includes an initial block, multiple graph convolutional neural network blocks, a random dropout layer, a global pooling layer and a fully connected layer connected in sequence; wherein the graph convolutional neural network block includes a spatial module and a temporal module; the spatial module includes an upper body hierarchical graph convolution operation; the temporal module includes four branches, corresponding to two dilated temporal convolutional layer branches, a point convolution branch and a maximum pooling branch; the dilated temporal convolutional layer branch includes a convolution layer, a batch normalization layer, an activation function and a temporal convolution layer; the point convolution branch includes a 1×1 convolution layer, a batch normalization layer and an activation function; the maximum pooling branch includes a 1×1 convolution layer, a batch normalization layer, an activation function, a maximum pooling layer and a batch normalization layer.

[0009] Furthermore, the spatial module includes five parallel convolutional layers for extracting correlations between human body joints.

[0010] Furthermore, before the joint flow data and the skeleton flow data are input into the multiple graph convolutional neural network models, it also includes: constructing a topological structure based on the joint flow data and the skeleton flow data to obtain an adjacency matrix; wherein the topological structure includes: decomposing the topological structure of the upper body of the driver to be detected into an N-ary tree, with the hip or the nose as the root node of the N-ary tree, and connecting all the nodes in all adjacent hierarchical node sets in the N-ary tree.

[0011] Furthermore, the multiple trained graph convolutional neural network models include: a graph convolutional neural network model trained using joint flow data with the hip as the root node, a graph convolutional neural network model trained using bone flow data with the hip as the root node, a graph convolutional neural network model trained using joint flow data with the nose as the root node, and a graph convolutional neural network model trained using bone flow data with the nose as the root node.

[0012] Furthermore, the classification probability of samples trained on multiple graph convolutional neural network models is calculated by the following normalized exponential function:

[0013]

[0014] The objective functions for training the multiple graph convolutional neural network models include:

[0015]

[0016] Among them, z is the output logical value of the fully connected layer, Q is the number of emotion categories, and z i represents the i-th element of z, is a one-hot vector representing the predicted value of the i-th emotion category.

[0017] On the other hand, a driver emotional state detection system based on upper body posture is also provided, and the system includes: an identification module, a preprocessing module, a detection module and a fusion module; wherein the identification module is used to identify the key parts of the human body in the upper body video clip of the driver to be detected based on a preset human posture estimation model, and obtain the time series of each joint position; the preprocessing module is used to preprocess the joint position time series and convert it into joint flow data and bone flow data; the detection module is used to input the joint flow data and the bone flow data into multiple trained graph convolutional neural network models respectively, and obtain multiple classification results; the fusion module is used to fuse the multiple classification results based on an integrated learning method to obtain the emotional state recognition result of the driver to be monitored.

[0018] Furthermore, the graph convolutional neural network model includes an initial block, multiple graph convolutional neural network blocks, a random dropout layer, a global pooling layer and a fully connected layer connected in sequence; wherein the graph convolutional neural network block includes a spatial module and a temporal module; the spatial module includes an upper body hierarchical graph convolution operation; the temporal module includes four branches, corresponding to two dilated temporal convolutional layer branches, a point convolution branch and a maximum pooling branch; the dilated temporal convolutional layer branch includes a convolution layer, a batch normalization layer, an activation function and a temporal convolution layer; the point convolution branch includes a 1×1 convolution layer, a batch normalization layer and an activation function; the maximum pooling branch includes a 1×1 convolution layer, a batch normalization layer, an activation function, a maximum pooling layer and a batch normalization layer.

[0019] Furthermore, the system also includes a construction module, which is used to: construct a topological structure based on the joint flow data and the bone flow data to obtain an adjacency matrix; wherein the topological structure includes: decomposing the topological structure of the upper body of the driver to be detected into an N-ary tree, with the hip or the nose as the root node of the N-ary tree, and connecting all the nodes in all adjacent hierarchical node sets in the N-ary tree.

[0020] On the other hand, a computer-readable storage medium is provided, in which a program code is stored. The program code can be called by a processor to execute the method provided by the present invention.

[0021] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: by constructing a topological relationship graph for the upper body posture of the human body and utilizing its graph convolutional neural network, it is possible to effectively extract action emotion state information; by integrating models trained under different graphs, it is possible to effectively aggregate feature information of different graph patterns, thereby further improving the accuracy of action emotion recognition in driving environments; by using a new loss function, relevant category information is introduced for network prediction, thereby alleviating the problem of imbalanced data category distribution to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a driver's emotional state detection method based on upper body posture provided by an embodiment of the present invention;

[0024] Figure 2 It is a schematic diagram of the overall architecture of a graph convolutional neural network model provided by an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of a topological structure constructed by an upper body posture of a driver to be detected provided by an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of the architecture of a space module provided by an embodiment of the present invention;

[0027] Figure 5 It is a schematic diagram of a driver emotional state detection system based on upper body posture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0029] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0030] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0031] Embodiment 1

[0032] Figure 1 FIG. 1 is a flow chart of a method for detecting a driver's emotional state based on upper body posture according to an embodiment of the present invention. Figure 1 As shown, the method specifically comprises the following steps:

[0033] Step S102 , identifying key parts of the human body in the upper body video clip of the driver to be detected based on a preset human posture estimation model, and obtaining a time series of each joint position.

[0034] Step S104, preprocessing the joint position time series and converting it into joint flow data and bone flow data.

[0035] Step S106, respectively inputting the joint flow data and the bone flow data into a plurality of trained graph convolutional neural network models to obtain a plurality of classification results.

[0036] Step S108, fusing multiple classification results based on an ensemble learning method to obtain an emotional state recognition result of the driver to be monitored.

[0037] Figure 2 Schematic diagram of the overall architecture of a graph convolutional neural network model provided according to an embodiment of the present invention. Figure 2 As shown, the graph convolutional neural network model includes an initial block (Initial), multiple graph convolutional neural network blocks, a random dropout layer (DropoutLayer), a global pooling layer and a fully connected layer (FCLayer) connected in sequence; wherein the graph convolutional neural network block includes a spatial module (UbH-GCN) and a temporal module (TCN);

[0038] The spatial module includes upper body hierarchical graph convolution operation (UbH-Graph convolution);

[0039] The temporal module consists of four branches, corresponding to two dilated temporal convolutional layer branches, one point convolution branch, and one maximum pooling branch;

[0040] Specifically, the dilated temporal convolutional layer branch includes a convolutional layer, a batch normalization layer, an activation function, and a temporal convolutional layer;

[0041] The point convolution branch consists of a 1×1 convolution layer, a batch normalization layer, and an activation function;

[0042] The max pooling branch consists of a 1×1 convolutional layer, a batch normalization layer, an activation function, a max pooling layer, and a batch normalization layer.

[0043] The spatial module consists of five parallel convolutional layers to extract the correlations between human joints.

[0044] Preferably, if Figure 2 As shown, the number of graph convolutional neural network blocks is 9, and the random activation rate of the random activation layer is 0.25.

[0045] Specifically, the method provided by the embodiment of the present invention, before inputting the joint flow data and the skeleton flow data into multiple graph convolutional neural network models, further includes:

[0046] Construct a topological structure based on joint stream data and bone stream data to obtain an adjacency matrix;

[0047] The topological structure includes: decomposing the topological structure of the upper body of the driver to be detected into an N-ary tree, taking the hip or the nose as the root node of the N-ary tree, and connecting all the nodes in all adjacent hierarchical node sets in the N-ary tree.

[0048] Figure 3 is a schematic diagram of a topological structure of the upper body posture of a driver to be detected according to an embodiment of the present invention. Figure 3 As shown, in order to construct such a topological structure, the present invention decomposes the natural topological structure of the human body into an N-ary tree, and allows nodes from different body parts to coexist in the same set through a determined root node. For example, the left wrist joint and the right wrist joint or the wrist joint and the ear joint can exist in the same node set. Once the root node is established, the present invention can convert the topological structure into an N-ary tree. The hierarchical structure of the tree allows nodes from different body parts to be classified, thereby extracting the hierarchical information of the graph. Therefore, the adjacency matrix Can be defined as:

[0049]

[0050] ε k =ε(H k ∪H k+1 ,H k →H k+1 ,Hk+1 →H k )

[0051] Among them, H k represents the k-th level node set, ε(H k →H k+1 ) indicates that from H k To H k+1 The edge set, N L and N H denote the number of levels in the hierarchy and the number of edges on the hierarchy boundary, respectively. k ) represents the set of five edge subsets s id ,s cp ,s cf , They represent identity, centripetal, centrifugal, two-hop centripetal and two-hop centrifugal edge subsets respectively.

[0052] The graph structure in the embodiment of the present invention connects all nodes in all adjacent hierarchical node sets to achieve the relationship between joints of different body parts. At the same time, in order to improve the adaptability and stability of training, we normalize the adjacency matrix into a degree matrix and use all elements in the matrix as learnable parameters.

[0053] Figure 4 FIG. 1 is a schematic diagram of the architecture of a space module provided according to an embodiment of the present invention. Figure 4 As shown, the UbH-GCN of the k-th hierarchical set contains five parallel convolutional layers, which are specially designed to extract the correlation between human joints. In order to reduce the computational complexity, all five branch operations use linear transformations. For these branches, the present invention performs subset-based graph convolution operations on hierarchical edge sets, and each edge set contains five edge subsets. In addition, the present invention also merges the result of the first branch operation into the results of the other four branches, not simply summing the output values ​​of the four branches, but connecting these output values ​​along the channel dimension. In order to enhance the learning ability of the network, the cascaded result is added to the input residual through a batch normalization layer. Finally, nonlinearity is introduced into the network structure through a linear rectification function layer.

[0054] Specifically, multiple trained graph convolutional neural network models include:

[0055] A graph convolutional neural network model trained using joint flow data with the hip as the root node, a graph convolutional neural network model trained using bone flow data with the hip as the root node, a graph convolutional neural network model trained using joint flow data with the nose as the root node, and a graph convolutional neural network model trained using bone flow data with the nose as the root node.

[0056] Specifically, the classification probability of samples trained on multiple graph convolutional neural network models is calculated using the following normalized exponential function:

[0057]

[0058] The objective functions for training multiple graph convolutional neural network models include:

[0059]

[0060] Among them, z is the output logical value of the fully connected layer, Q is the number of emotion categories, and z i represents the i-th element of z, is a one-hot vector representing the predicted value of the i-th emotion category.

[0061] The method provided by an embodiment of the present invention uses a new integrated learning method, which only uses joint and bone streams without using any motion streams. Specifically, the present invention uses two skeleton graphs at different levels, each for training joint streams and bone streams. The two graphs have different root nodes - hip and nose key points. That is, the present invention uses a hierarchical skeleton graph with the hip as the root node to train the joint and bone streams, and when the root node is the nose, the present invention also uses a similar graph for training. Since the models of the two different graphs need to be trained in different aspects, each graph consists of a different set of relationship edges. For example, if the root node is the hip, the hip key point is only connected to the chest key point, and when the root node is the nose, the hip key point is connected to most of the upper body key points.

[0062] Due to the imbalanced number of samples in different categories, the features of the tail categories may be squeezed into a narrow area in the feature space. In order to achieve a balanced feature distribution, the present invention introduces category changes for network predictions during the training phase, so that an instance is no longer projected onto a feature point, but onto a small area. This perturbation depends heavily on the category scale, manifesting as assigning smaller changes to the head category and larger changes to the tail category.

[0063] Assume that the output logical value of the last fully connected layer of the model is defined as Where Q is the number of emotion categories. The classification probability of the input sample can be calculated by the normalized exponential function:

[0064] Among them, z i represents the element of the ith z. Then, the cross entropy loss function is as follows, and the present invention adjusts it accordingly:

[0065]

[0066] in, is a one-hot vector representing the predicted value of the i-th emotion category. The derivation process of the above two formulas shows the basic relationship between the cross entropy loss function and the logical value. It should be noted that the dimension of the logical value z corresponds to the total number of categories and directly affects the calculation of the loss, making it the most intuitive factor affecting the size of the classification feature space.

[0067]

[0068] Among them, q j is the number of instances with category j, and δ is a Gaussian distribution with mean 0 and standard deviation σ. The above formula adjusts the variability within each category by processing categories based on differences in the number of instances. It assigns smaller changes to categories with more instances and larger changes to categories with fewer instances. The method provided by an embodiment of the present invention expands the representation of each instance in the feature space from a single point to a certain range of regions, which helps to balance inter-class representations during training. In addition, to ensure that the predictions in the inference phase are reliable, the changes introduced during training are excluded.

[0069] From the above description, it can be seen that the embodiment of the present invention provides a method for detecting the driver's emotional state based on upper body posture, which uses the upper body skeleton sequence to overcome the limitation of full body posture capture in driving scenarios; the architecture adopts the upper body hierarchical graph (UbH-Graph) to dynamically capture the relationship between upper body movement and emotional state; class-specific changes are uniquely incorporated during the training process to balance feature distribution and enhance emotion recognition. The method provided by the embodiment of the present invention surpasses the existing multimodal methods on the assisted driving dataset and demonstrates robustness and adaptability on the daily action dataset.

[0070] Embodiment 2

[0071] Figure 5 is a schematic diagram of a driver emotional state detection system based on upper body posture according to an embodiment of the present invention. Figure 5 As shown, the system includes: a recognition module 10, a preprocessing module 20, a detection module 30 and a fusion module 40.

[0072] Specifically, the recognition module 10 is used to recognize key human body parts in the upper body video clip of the driver to be detected based on a preset human body posture estimation model, and obtain a time series of each joint position.

[0073] The preprocessing module 20 is used to preprocess the joint position time series and convert it into joint flow data and bone flow data.

[0074] The detection module 30 is used to input the joint flow data and the bone flow data into multiple trained graph convolutional neural network models respectively to obtain multiple classification results.

[0075] The fusion module 40 is used to fuse multiple classification results based on an integrated learning method to obtain an emotional state recognition result of the driver to be monitored.

[0076] Specifically, the graph convolutional neural network model includes an initial block, multiple graph convolutional neural network blocks, a random dropout layer, a global pooling layer, and a fully connected layer connected in sequence; wherein the graph convolutional neural network block includes a spatial module and a temporal module;

[0077] The spatial module includes the upper body hierarchical graph convolution operations;

[0078] The temporal module consists of four branches, corresponding to two dilated temporal convolutional layer branches, one point convolution branch and a maximum pooling branch; the dilated temporal convolutional layer branch includes a convolution layer, a batch normalization layer, an activation function and a temporal convolution layer; the point convolution branch includes a 1×1 convolution layer, a batch normalization layer and an activation function; the maximum pooling branch includes a 1×1 convolution layer, a batch normalization layer, an activation function, a maximum pooling layer and a batch normalization layer.

[0079] Specifically, the spatial module consists of five parallel convolutional layers to extract the correlations between human joints.

[0080] Specifically, multiple trained graph convolutional neural network models include:

[0081] A graph convolutional neural network model trained using joint flow data with the hip as the root node, a graph convolutional neural network model trained using bone flow data with the hip as the root node, a graph convolutional neural network model trained using joint flow data with the nose as the root node, and a graph convolutional neural network model trained using bone flow data with the nose as the root node.

[0082] The classification probability of samples trained on multiple graph convolutional neural network models is calculated using the following normalized exponential function:

[0083]

[0084] The objective functions for training multiple graph convolutional neural network models include:

[0085]

[0086] Among them, z is the output logical value of the fully connected layer, Q is the number of emotion categories, and z i represents the i-th element of z, is a one-hot vector representing the predicted value of the i-th emotion category.

[0087] Specifically, Figure 5 As shown, the system further includes a building block 50 for:

[0088] Construct a topological structure based on joint stream data and bone stream data to obtain an adjacency matrix;

[0089] The topological structure includes: decomposing the topological structure of the upper body of the driver to be detected into an N-ary tree, taking the hip or the nose as the root node of the N-ary tree, and connecting all the nodes in all adjacent hierarchical node sets in the N-ary tree.

[0090] The present invention further provides a computer-readable storage medium, in which program codes are stored. The program codes can be called by a processor to execute the method in the above-mentioned embodiment 1.

[0091] It should be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0092] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0093] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0094] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0095] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0096] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0097] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0099] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0100] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for detecting a driver's emotional state based on upper body posture, characterized in that: The method comprises: Based on a preset human posture estimation model, key human body parts in the upper body video clip of the driver to be detected are identified to obtain a time series of each joint position; Preprocessing the joint position time series to convert it into joint stream data and bone stream data; Inputting the joint flow data and the bone flow data into a plurality of trained graph convolutional neural network models respectively to obtain a plurality of classification results; The graph convolutional neural network model includes an initial block, a plurality of graph convolutional neural network blocks, a random dropout layer, a global pooling layer and a fully connected layer connected in sequence; wherein the graph convolutional neural network block includes a spatial module and a temporal module; The spatial module includes an upper body hierarchical graph convolution operation; The time module includes four branches, which correspond to two dilated time convolution layer branches, a point convolution branch and a maximum pooling branch respectively; the dilated time convolution layer branch includes a convolution layer, a batch normalization layer, an activation function and a time convolution layer; the point convolution branch includes a 1×1 convolution layer, a batch normalization layer and an activation function; the maximum pooling branch includes a 1×1 convolution layer, a batch normalization layer, an activation function, a maximum pooling layer and a batch normalization layer; The spatial module includes five parallel convolutional layers for extracting the correlation between human body joints; The multiple classification results are fused based on an integrated learning method to obtain an emotional state recognition result of the driver to be detected.

2. The method according to claim 1, characterized in that Before inputting the joint stream data and the skeleton stream data into the multiple graph convolutional neural network models, the method further includes: Building a topological structure based on the joint stream data and the skeleton stream data to obtain an adjacency matrix; The topological structure includes: decomposing the topological structure of the upper body of the driver to be detected into an N-ary tree, taking the hip or the nose as the root node of the N-ary tree, and connecting all the nodes in all adjacent hierarchical node sets in the N-ary tree.

3. The method according to claim 2, characterized in that The multiple trained graph convolutional neural network models include: A graph convolutional neural network model trained using joint flow data with the hip as the root node, a graph convolutional neural network model trained using bone flow data with the hip as the root node, a graph convolutional neural network model trained using joint flow data with the nose as the root node, and a graph convolutional neural network model trained using bone flow data with the nose as the root node.

4. The method according to claim 1, characterized in that: The classification probability of samples trained on multiple graph convolutional neural network models is calculated using the following normalized exponential function: The objective functions for training the multiple graph convolutional neural network models include: Among them, z is the output logical value of the fully connected layer, Q is the number of emotion categories, and z i represents the i-th element of z, is a one-hot vector representing the predicted value of the i-th emotion category.

5. A driver emotional state detection system based on upper body posture, characterized in that: The system includes: a recognition module, a preprocessing module, a detection module and a fusion module; wherein, The recognition module is used to recognize key parts of the human body in the upper body video clip of the driver to be detected based on a preset human posture estimation model, and obtain a time series of each joint position; The preprocessing module is used to preprocess the joint position time series and convert it into joint flow data and bone flow data; The detection module is used to input the joint flow data and the bone flow data into a plurality of trained graph convolutional neural network models respectively to obtain a plurality of classification results; The graph convolutional neural network model includes an initial block, a plurality of graph convolutional neural network blocks, a random dropout layer, a global pooling layer and a fully connected layer connected in sequence; wherein the graph convolutional neural network block includes a spatial module and a temporal module; The spatial module includes an upper body hierarchical graph convolution operation; The time module includes four branches, which correspond to two dilated time convolution layer branches, a point convolution branch and a maximum pooling branch respectively; the dilated time convolution layer branch includes a convolution layer, a batch normalization layer, an activation function and a time convolution layer; the point convolution branch includes a 1×1 convolution layer, a batch normalization layer and an activation function; the maximum pooling branch includes a 1×1 convolution layer, a batch normalization layer, an activation function, a maximum pooling layer and a batch normalization layer; The spatial module includes five parallel convolutional layers for extracting the correlation between human body joints; The fusion module is used to fuse the multiple classification results based on an integrated learning method to obtain the emotional state recognition result of the driver to be detected.

6. The system according to claim 5, characterized in that The system also includes a building block for: Building a topological structure based on the joint stream data and the skeleton stream data to obtain an adjacency matrix; The topological structure includes: decomposing the topological structure of the upper body of the driver to be detected into an N-ary tree, taking the hip or the nose as the root node of the N-ary tree, and connecting all the nodes in all adjacent hierarchical node sets in the N-ary tree.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 4.

Images