Emotion detection and response system and method based on driving position

Abstract

The invention discloses an emotion detection and response system and method based on a driving seat, and relates to the technical field of image analysis. Face region recognition is carried out on the face image, and then face alignment is carried out to obtain a detection image; substituting the detection image into a feature point detection model for feature point detection to obtain a face feature point set, and performing real-time face tracking on the driver according to the face feature point set to obtain a tracking image; and performing emotion analysis on the tracking image on the time sequence, and generating a coping scheme according to an emotion analysis result. Facial images are recognized and aligned through a spatial transformation network model, it is ensured that extracted facial features are more accurate, then facial feature points are accurately positioned through a feature point detection model to obtain a tracking image, emotion analysis is conducted on the tracking image, and therefore the current emotion state of a driver is quickly recognized, and the driver experience is improved. And the real-time performance and accuracy of the system in emotion detection and response are improved.

Description

Technical Field

[0001] The present invention belongs to the field of image analysis technology, and specifically relates to an emotion detection and response system and method based on a driver's seat Background Art

[0002] With the rapid expansion of the number of vehicles worldwide, the frequency of traffic accidents has also shown an upward trend that cannot be ignored. This situation seriously threatens road safety and has become a global safety issue that needs to be solved urgently. The transportation system is a complex network composed of road users, various vehicles and road infrastructure. Every link of its operation is closely connected, and the occurrence of traffic accidents is often due to the interweaving and superposition of multiple factors.

[0003] Among these many reasons, the impact of the driver's emotional state on driving behavior is particularly prominent and has become a key factor that cannot be ignored. Driving, as a task that requires high concentration and immediate response, is easily distracted by various stimuli from the external environment. The driver's emotional state at the time, whether positive or negative, will directly or indirectly affect their driving behavior and decision-making process.

[0004] It is particularly noteworthy that when drivers are in a negative emotional state such as anger or anxiety, they often exhibit aggressive and unstable driving behaviors, such as speeding, frequent lane changes, and ignoring traffic rules, which greatly increases the risk of traffic accidents. Existing driver emotion detection often requires a lot of time to analyze, and the accuracy of the analysis is not high, which makes the system's real-time and accuracy in emotion detection and response low. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the existing driver emotion detection often requires a lot of time for analysis and the accuracy of the analysis is not high, which makes the system's real-time and accuracy in emotion detection and response low, and to propose an emotion detection and response system and method based on the driver's seat.

[0006] In a first aspect of the present invention, a method for detecting and coping with emotions based on a driver's seat is first proposed, the method comprising:

[0007] Collecting the driver's facial video data in real time, and performing frame extraction processing on the facial video data through a first preset time window to obtain a facial image set in a time series;

[0008] Performing face region recognition on a target face image to obtain a target region image, and inputting the target region image into a spatial transformation network model to perform face alignment to obtain a detection image; the target face image is any one face image in the face image set;

[0009] Bringing the detection image into a feature point detection model to perform feature point detection to obtain a facial feature point set, performing real-time facial tracking on the driver to obtain a tracking image based on the facial feature point set, and adjusting the first preset time window based on the tracking image in a time series;

[0010] Perform sentiment analysis on the tracking images in time series and generate response plans based on the sentiment analysis results.

[0011] Optionally, performing face region recognition on the target facial image to obtain a target region image includes:

[0012] Performing data processing on the target facial image to obtain first data, and performing multi-feature extraction on the first data through central difference convolution to obtain multiple initial features;

[0013] Multiple first features are obtained by performing spatial attention enhancement on each initial feature through multi-scale feature modulation;

[0014] For each first feature, performing center difference feature extraction on the first feature through depth-wise separable center difference convolution to obtain a second feature;

[0015] The target feature is obtained by fusing all the second features through multi-scale channel attention;

[0016] A face region is identified through a region proposal network according to the target feature, and the target facial image is cut according to the face region to obtain a target region image.

[0017] Optionally, inputting the target area image into a spatial transformation network model to perform face alignment to obtain a detection image includes:

[0018] Extracting affine transformation parameters of the target area image to obtain target affine parameters, and obtaining an affine transformation matrix according to the target affine parameters;

[0019] Extracting the pixel value corresponding to each pixel in the target area image by bilinear interpolation;

[0020] Performing pixel inverse mapping on the pixel value corresponding to each pixel according to the affine transformation matrix to obtain a mapping image;

[0021] The target feature is mapped onto the mapping image, and convolution is performed through a convolution layer to obtain a detection image.

[0022] Optionally, performing real-time facial tracking on the driver to obtain a tracking image according to the facial feature point set includes:

[0023] Acquire the detection image in real time, obtain the feature points of the current image, and calculate the position change of each feature point between the current image and the previous detection image through Euclidean distance;

[0024] If there is a pixel point whose position change is greater than the preset change value, the pixel point is recorded as the target pixel point, and the pixel detection area is obtained with the target pixel point as the center and the preset length as the radius;

[0025] The target ratio of the target pixel points to the feature points in the pixel detection area is counted, and if the target ratio is greater than a preset ratio value, the detection image is recorded as a tracking image.

[0026] Optionally, sentiment analysis of the tracking images over time series includes:

[0027] Bringing the tracking image into the residual network to extract feature information, and activating the extracted feature information through Gumbel Softmax to obtain an activation signal;

[0028] Bringing the activation signal and the tracking image into the working memory network and the long-term memory network respectively to obtain a first memory feature and a second memory feature, and obtaining a final fusion feature according to the first memory feature and the second memory feature;

[0029] The final fusion feature is brought into a preset database for emotion similarity comparison to obtain the emotion analysis result; the preset database is used to store the features corresponding to different emotions in different combinations.

[0030] In a second aspect of the present invention, a driver-side-based emotion detection and response system is proposed, comprising:

[0031] A facial image acquisition module, used for collecting the driver's facial video data in real time, and performing frame extraction processing on the facial video data through a first preset time window to obtain a facial image set in a time series;

[0032] A face alignment module is used to perform face region recognition on a target face image to obtain a target region image, and input the target region image into a spatial transformation network model to perform face alignment to obtain a detection image; the target face image is any one of the face images in the face image set;

[0033] A face tracking module, used for bringing the detection image into a feature point detection model to perform feature point detection to obtain a facial feature point set, performing real-time face tracking on the driver to obtain a tracking image based on the facial feature point set, and adjusting the first preset time window based on the tracking image in a time series;

[0034] The sentiment analysis module is used to perform sentiment analysis on the tracking images in time series and generate response plans based on the sentiment analysis results.

[0035] Optionally, the face alignment includes:

[0036] A multi-feature extraction module, used for performing data processing on the target facial image to obtain first data, and performing multi-feature extraction on the first data through central difference convolution to obtain multiple initial features;

[0037] An attention enhancement module, configured to perform spatial attention enhancement on each initial feature through multi-scale feature modulation to obtain a plurality of first features;

[0038] A center difference feature extraction module, used for performing center difference feature extraction on each first feature through a depth-separable center difference convolution to obtain a second feature;

[0039] A feature fusion module is used to fuse all the second features to obtain the target feature through multi-scale channel attention;

[0040] The face region recognition module is used to recognize the face region through a region proposal network according to the target features, and to cut the target facial image according to the face region to obtain a target region image.

[0041] Optionally, the face alignment further includes:

[0042] An affine parameter extraction module is used to extract affine transformation parameters from the target area image to obtain target affine parameters, and obtain an affine transformation matrix according to the target affine parameters;

[0043] A pixel value extraction module, used to extract the pixel value corresponding to each pixel in the target area image through bilinear interpolation;

[0044] A pixel mapping module, used for performing pixel inverse mapping on the pixel value corresponding to each pixel according to the affine transformation matrix to obtain a mapping image;

[0045] The image convolution module is used to map the target features onto the mapping image and obtain a detection image by convolution through a convolution layer.

[0046] Optionally, the face tracking module includes:

[0047] The position change calculation module is used to obtain the detection image in real time, obtain the feature points of the current image, and calculate the position change of each feature point between the current image and the previous detection image through the Euclidean distance;

[0048] A pixel detection area determination module is used to record a pixel point as a target pixel point if the position change of a pixel point is greater than a preset change value, and obtain a pixel detection area with the target pixel point as the center and a preset length as the radius;

[0049] The tracking image confirmation module is used to count the target ratio of target pixels and feature points in the pixel detection area. If the target ratio is greater than a preset ratio value, the detection image is recorded as a tracking image.

[0050] Optionally, the sentiment analysis module includes:

[0051] An activation signal generating module, used for bringing the tracking image into the residual network to extract feature information, and activating the extracted feature information through Gumbel Softmax to obtain an activation signal;

[0052] A final fusion feature generation module is used to bring the activation signal and the tracking image into the working memory network and the long-term memory network respectively to obtain a first memory feature and a second memory feature, and obtain a final fusion feature according to the first memory feature and the second memory feature;

[0053] The emotion comparison module is used to bring the final fusion feature into a preset database for emotion similarity comparison to obtain emotion analysis results; the preset database is used to store features corresponding to different emotions in different combinations.

[0054] Beneficial effects of the present invention:

[0055] The present invention proposes an emotion detection and response method based on a driving position, which collects the driver's facial video data in real time, performs frame extraction processing on the facial video data through a first preset time window to obtain a facial image set in a time series; performs face area recognition on a target facial image to obtain a target area image, and inputs the target area image into a spatial transformation network model to perform face alignment to obtain a detection image; the target facial image is any facial image in the facial image set; the detection image is brought into a feature point detection model to perform feature point detection to obtain a facial feature point set, and based on the facial feature point set, the driver's face is tracked in real time to obtain a tracking image, and the first preset time window is adjusted based on the tracking image in the time series; emotion analysis is performed on the tracking image in the time series, and a response plan is generated based on the emotion analysis result. By monitoring the driver's facial expressions in real time, we ensure that the driver's emotional state can be obtained instantly. We use the spatial transformation network model to identify and align facial images to ensure that the extracted facial features are more accurate. Then, we use the feature point detection model to accurately locate the facial feature points, and use these feature points to track the face, which further improves the accuracy of emotion recognition and the system's response speed to facial changes. By performing emotion analysis on the processed tracking image, we can quickly identify the driver's current emotional state, improving the real-time and accuracy of the system in emotion detection and response. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention will be further described below in conjunction with the accompanying drawings.

[0057] Figure 1 A flowchart of a method for detecting and coping with emotions based on a driver's seat is provided for an embodiment of the present invention.

[0058] Figure 2 A framework diagram of a driver-side-based emotion detection and response system is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the present invention, the description of "first", "second", etc. is only used for descriptive purposes, and cannot be understood as indicating or implying its relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0060] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0061] The embodiment of the present invention provides a method for detecting and coping with emotions based on the driver's seat. Figure 1 , Figure 1 A flowchart of a method for detecting and coping with emotions based on a driver's seat provided in an embodiment of the present invention. The method comprises the following steps:

[0062] S101, collecting facial video data of the driver in real time, and performing frame extraction processing on the facial video data through a first preset time window to obtain a facial image set in a time series;

[0063] S102, performing face region recognition on the target face image to obtain a target region image, and inputting the target region image into a spatial transformation network model to perform face alignment to obtain a detection image; the target face image is any face image in the face image set;

[0064] S103, bringing the detection image into a feature point detection model to perform feature point detection to obtain a facial feature point set, performing real-time face tracking on the driver according to the facial feature point set to obtain a tracking image, and adjusting the first preset time window according to the tracking image in the time series;

[0065] S104, performing sentiment analysis on the tracking images in the time series, and generating a response plan based on the sentiment analysis results.

[0066] An emotion detection and response method based on a driver's seat provided by an embodiment of the present invention ensures that the driver's emotional state can be obtained instantly by monitoring the driver's facial expressions in real time, and recognizes and aligns facial images through a spatial transformation network model to ensure that the extracted facial features are more accurate. The facial feature points are accurately located by using a feature point detection model, and the face is tracked through these feature points, thereby further improving the accuracy of emotion recognition and the system's response speed to facial changes. Emotion analysis is then performed on the processed tracking image to quickly identify the driver's current emotional state, thereby improving the real-time and accuracy of the system in emotion detection and response.

[0067] In one implementation, by collecting and analyzing the driver's facial video data in real time, the system can quickly identify the driver's emotional state and generate a response plan in a timely manner when negative emotions such as anger, anxiety or fatigue are detected, thereby reducing traffic accidents caused by emotional problems and improving overall road safety.

[0068] In one implementation, the system continuously collects and processes facial video data to ensure immediate response to the driver's emotional changes. The efficient implementation of feature point detection and face tracking makes emotional analysis faster and ensures the real-time performance of the system.

[0069] In one implementation, the emotion analysis results include anger, anxiety, happiness, sadness, etc. For example, when the analysis result is anger, a warning pops up to remind the driver to remain calm, suggest a short stop and rest, increase warnings on vehicle speed, vehicle distance, etc., remind the driver to pay attention to safety, and functions such as automatic braking and lane keeping can intervene more actively; if the analysis result is sadness, comfort is provided through friendly voice prompts to encourage the driver to remain calm and focus, and some comforting songs can be played; if the analysis result is happiness, moderate safety reminders are provided to prevent driving negligence caused by excessive relaxation; if the analysis result is anxiety, soothing music or white noise is played to help the driver relax and reduce anxiety, and clear and detailed navigation guidance is provided.

[0070] In one implementation, a spatial transformation network is used for face alignment, and a feature point detection model is used for detailed feature point positioning, which improves the accuracy of facial image processing. Accurate face tracking and emotion analysis can more reliably reflect the driver's emotional state.

[0071] In one implementation, the first preset time window is determined by a technician. For example, an image is first extracted every 10 frames to obtain a detection image, and the detection image is analyzed to determine whether it is a tracking image. If not, the detection image corresponding to the next preset time window is calculated until a tracking image appears. The time distance between two adjacent tracking images is calculated to obtain a second time window, and the first preset time window is updated according to the average value of the first preset time window and the second time window.

[0072] In one implementation, the tracking image is determined by performing feature point position transformation analysis on the detection images in a time series.

[0073] In one implementation, the feature point detection model can be Dlib's Facial LandmarkDetector, OpenCV's Facemark API, deep learning-based models such as MTCNN, MobileNet, and BlazeFace, etc.

[0074] In one embodiment, performing face region recognition on a target facial image to obtain a target region image includes:

[0075] Performing data processing on the target facial image to obtain first data, and performing multi-feature extraction on the first data through central difference convolution to obtain multiple initial features;

[0076] Multiple first features are obtained by performing spatial attention enhancement on each initial feature through multi-scale feature modulation;

[0077] For each first feature, performing center difference feature extraction on the first feature through depth-wise separable center difference convolution to obtain a second feature;

[0078] The target feature is obtained by fusing all the second features through multi-scale channel attention;

[0079] According to the target features, the face area is identified through the region proposal network, and the target facial image is cut according to the face area to obtain the target area image.

[0080] In one implementation, center difference convolution and depthwise separable center difference convolution are used for multi-feature extraction and feature enhancement, which improves the accuracy and detail of feature extraction and ensures that the extracted facial feature information is richer and more accurate.

[0081] In one implementation, data processing is to remove noise, errors, duplications or missing data, and perform data enhancement such as rotation, scaling, etc.

[0082] In one implementation, the first data is processed with multiple convolution kernels. For example, four 3x3 convolution kernels, four 5x5 convolution kernels, and four 7x7 convolution kernels are used to perform horizontal and vertical convolutions on the first data, respectively, to generate horizontal and vertical gradient maps of the image. The images generated by the horizontal and vertical convolutions corresponding to each convolution kernel are fused to obtain the initial features.

[0083] In one implementation, for each initial feature, feature maps of different scales are generated through convolution operations of different scales, the weight of each feature map is calculated through the spatial attention mechanism, the calculated weight is applied to the feature maps of different scales and then fused to obtain the first feature.

[0084] In one implementation, the face area is identified according to the target features through the region proposal network. Specifically, a convolutional neural network is used to extract the target features to obtain the features of each position of the facial image, and a regression network is used to predict the offset of the bounding box. At the same time, each position is classified to determine whether the position contains the target object. The RPN (region proposal network) uses predefined anchor boxes of different sizes and aspect ratios as a reference. Each anchor box is associated with a position in the feature map. According to the bounding box regression output of the RPN, the position of the anchor box is adjusted to generate a new candidate region. Non-maximum suppression is applied to the generated candidate region, and repeated or overlapping proposals are removed. The region most likely to contain the target is retained to obtain the face region.

[0085] In one implementation, spatial attention is enhanced for each initial feature through multi-scale feature modulation, so that the model can focus on important areas in the image and ignore irrelevant information, thereby improving the model's ability to recognize features of different scales; spatial attention enhancement is performed after initial feature extraction, which helps the model better capture key information in facial images, enhance the representation ability of features, and improve the effect of face recognition.

[0086] In one implementation, all second features are fused through multi-scale channel attention to further improve the accuracy and robustness of feature representation, making the final target features more comprehensive and accurate; the face area is identified using the region proposal network (RPN), so that the system can still accurately locate the face area when dealing with complex backgrounds and diverse facial postures, reducing the probability of false detection and missed detection.

[0087] In one implementation, efficient convolution operations such as depthwise separable convolution are used to reduce computational complexity and improve processing speed, so that the system can achieve real-time processing while ensuring high accuracy, thereby realizing efficient face recognition and emotion detection in practical applications.

[0088] In one embodiment, inputting the target area image into the spatial transformation network model to perform face alignment to obtain a detection image includes:

[0089] Extracting affine transformation parameters of the target area image to obtain target affine parameters, and obtaining an affine transformation matrix according to the target affine parameters;

[0090] Extract the pixel value corresponding to each pixel in the target area image through bilinear interpolation;

[0091] Perform pixel reverse mapping on the pixel value corresponding to each pixel according to the affine transformation matrix to obtain a mapping image;

[0092] The target features are mapped to the mapping image and convolved through the convolution layer to obtain the detection image.

[0093] In one implementation, by extracting the affine transformation parameters of the target area image and calculating the affine transformation matrix, precise alignment of the image can be achieved, which helps to accurately perform geometric correction on the target area, eliminate distortion and asymmetry in the image, and thus improve the accuracy of subsequent processing.

[0094] In one implementation, bilinear interpolation technology is used to extract pixel values ​​from the target area image, which can effectively resample the image and estimate the value of each new pixel by considering the weighted average of neighboring pixels, thereby reducing artifacts and blur that may occur during the interpolation process and improving the image quality and detail retention.

[0095] In one implementation, the model order for affine transformation parameter extraction is: convolution layer with 32 5x5 convolution kernels, step size 1, activation function ReLu; maximum pooling layer: 2x2 window, step size 2; convolution layer: 64 5x5 convolution kernels, step size 1, activation function ReLu; maximum pooling layer: 2x2 window, step size 2, fully connected layer: 128 neurons, activation function ReLu; the fully connected layer outputs 6 neurons recorded as the target affine parameters.

[0096] In one implementation, each pixel is reverse mapped according to the affine transformation matrix, which can ensure the correct position of each pixel in the target image, so that each pixel of the image can be accurately mapped to a new position, avoiding information loss and position errors, and ensuring the accuracy of image transformation.

[0097] In one implementation, after affine transformation and resampling, the target features are mapped to the mapping image to ensure that the feature information is consistent with the geometric structure of the image. The mapping image is convolved by a convolution layer to further extract and enhance the feature information, supporting more accurate target detection and recognition.

[0098] In one embodiment, performing real-time face tracking on the driver to obtain a tracking image according to the facial feature point set includes:

[0099] Acquire the detection image in real time, obtain the feature points of the current image, and calculate the position change of each feature point between the current image and the previous detection image through Euclidean distance;

[0100] If there is a pixel point whose position change is greater than the preset change value, the pixel point is recorded as the target pixel point, and the pixel detection area is obtained with the target pixel point as the center and the preset length as the radius;

[0101] The target ratio of target pixels and feature points in the pixel detection area is counted. If the target ratio is greater than the preset ratio value, the detection image is recorded as a tracking image.

[0102] In one implementation, the position change of feature points is calculated by Euclidean distance, which can accurately measure the displacement of each feature point in the image, help detect the movement of the target object, and ensure the accuracy of tracking.

[0103] In one implementation, if the position change of a pixel exceeds a preset change value, it is marked as a target pixel, which can effectively identify areas in the image where significant changes have occurred, and help to accurately locate targets in complex environments.

[0104] In one implementation, a pixel detection area is obtained with the target pixel as the center and a preset length as the radius, ensuring that the target pixel is analyzed intensively within a specific area, reducing interference from other pixels and improving detection accuracy.

[0105] In one implementation, the preset change value, preset length, and preset ratio value are determined by a technician. For example, the preset change value is 5 pixels, the preset length is 25 pixels, and the preset ratio value is 60%. The position change of each pixel is calculated. If it is greater than 5 pixels, a circle is made with the pixel as the center and 25 pixels as the radius to obtain a pixel detection area. The position change of each feature point in the pixel detection area corresponding to the pixel is calculated. If it exists, the feature point is recorded as a target pixel point. The number of target pixels and the number of feature points in the pixel detection area are counted, and the target ratio is obtained according to the ratio of the number of target pixels to the number of feature points. If the target ratio is greater than 60%, the detected image is recorded as a tracking image.

[0106] In one embodiment, performing sentiment analysis on the tracking images in time series includes:

[0107] The tracking image is brought into the residual network to extract feature information, and the extracted feature information is activated through Gumbel Softmax to obtain an activation signal;

[0108] The activation signal and the tracking image are respectively brought into the working memory network and the long-term memory network to obtain the first memory feature and the second memory feature, and the final fusion feature is obtained according to the first memory feature and the second memory feature;

[0109] The final fusion features are brought into the preset database for emotion similarity comparison to obtain the emotion analysis results; the preset database is used to store the features corresponding to different emotions in different combinations.

[0110] In one implementation, the residual network can effectively extract high-level features in the image through its deep network structure, avoiding the gradient vanishing problem. Gumbel Softmax is a smooth discretization method that can effectively activate features while retaining gradient information.

[0111] In one implementation, the working memory network can store and utilize recently input feature information to effectively memorize and process real-time changes; the long-term memory network can preserve long-term feature information and provide the ability to recognize long-term emotional patterns.

[0112] In one implementation, Gumbel noise is generated, and Gumbel noise is added to the extracted feature information to obtain the input of Gumbel Softmax, and then the Softmax function is used to calculate the activation probability to obtain the probability distribution, and the probability distribution is recorded as the activation function.

[0113] In one implementation, a preset database contains a set of feature vectors under different emotional states. These feature vectors are obtained through the above-mentioned scheme during the training phase, and represent typical characteristics of various emotional states such as happiness, sadness, anger, etc. Each emotional state has a corresponding feature vector stored in the database. The similarity between the final fused feature and the feature in the database is calculated by cosine similarity, and the emotional state with the highest similarity is selected as the analysis result.

[0114] In one implementation, the working memory network includes a plurality of attention-integrated residual modules, and different residual modules in the working memory network are activated by activation signals to obtain a first residual module set, and a tracking image is subjected to feature extraction according to the first residual module set to obtain a first memory feature; the long-term memory network includes a plurality of attention-integrated residual modules, and different residual modules in the working memory network are activated by activation signals to obtain a second residual module set, and a tracking image is subjected to feature extraction according to the second residual module set to obtain a second memory feature, and the first memory feature and the second memory feature are superimposed to obtain a final fusion feature.

[0115] In one implementation, high-precision emotion analysis can be achieved by comparing the final fused features with the features in a preset database. The preset database stores the features of different emotions under different conditions, so that the system can accurately match and identify the current emotion.

[0116] In one implementation, by combining the first memory feature and the second memory feature, short-term and long-term feature information can be comprehensively considered, and the final fusion feature provides a comprehensive emotion representation, making emotion recognition more accurate and comprehensive.

[0117] Based on the same inventive concept, the embodiment of the present invention also provides an emotion detection and response system based on the driver's seat. Figure 2 , Figure 2 A framework diagram of an emotion detection and response system based on a driver's seat provided in an embodiment of the present invention includes:

[0118] A facial image acquisition module is used to collect the driver's facial video data in real time, and extract frames from the facial video data through a first preset time window to obtain a facial image set in a time series;

[0119] A face alignment module is used to perform face region recognition on a target face image to obtain a target region image, and input the target region image into a spatial transformation network model to perform face alignment to obtain a detection image; the target face image is any face image in a face image set;

[0120] A face tracking module, used for bringing the detection image into a feature point detection model to perform feature point detection to obtain a facial feature point set, performing real-time face tracking on the driver to obtain a tracking image based on the facial feature point set, and adjusting the first preset time window based on the tracking image in the time series;

[0121] The sentiment analysis module is used to perform sentiment analysis on the tracking images in time series and generate response plans based on the sentiment analysis results.

[0122] An emotion detection and response system based on a driver's seat provided by an embodiment of the present invention ensures that the driver's emotional state can be instantly acquired by monitoring the driver's facial expressions in real time, recognizes and aligns facial images through a spatial transformation network model to ensure that the extracted facial features are more accurate, and then accurately locates facial feature points by using a feature point detection model, and tracks the face through these feature points, thereby further improving the accuracy of emotion recognition and the system's response speed to facial changes, and then performing emotion analysis on the processed tracking image to quickly identify the driver's current emotional state, thereby improving the real-time and accuracy of the system in emotion detection and response.

[0123] In one embodiment, face alignment includes:

[0124] A multi-feature extraction module, used for performing data processing on the target facial image to obtain first data, and performing multi-feature extraction on the first data through central difference convolution to obtain multiple initial features;

[0125] An attention enhancement module, configured to perform spatial attention enhancement on each initial feature through multi-scale feature modulation to obtain a plurality of first features;

[0126] A center difference feature extraction module, used for performing center difference feature extraction on each first feature through a depth-separable center difference convolution to obtain a second feature;

[0127] A feature fusion module is used to fuse all the second features to obtain the target feature through multi-scale channel attention;

[0128] The face region recognition module is used to identify the face region through the region proposal network according to the target features, and to cut the target facial image according to the face region to obtain the target region image.

[0129] In one embodiment, face alignment further includes:

[0130] An affine parameter extraction module is used to extract affine transformation parameters of the target area image to obtain target affine parameters, and obtain an affine transformation matrix according to the target affine parameters;

[0131] A pixel value extraction module is used to extract the pixel value corresponding to each pixel in the target area image through bilinear interpolation;

[0132] A pixel mapping module, used for performing pixel inverse mapping on the pixel value corresponding to each pixel according to the affine transformation matrix to obtain a mapping image;

[0133] The image convolution module is used to map the target features to the mapping image and obtain the detection image through convolution layer.

[0134] In one embodiment, the face tracking module includes:

[0135] The position change calculation module is used to obtain the detection image in real time, obtain the feature points of the current image, and calculate the position change of each feature point between the current image and the previous detection image through the Euclidean distance;

[0136] A pixel detection area determination module is used to record a pixel point as a target pixel point if the position change of a pixel point is greater than a preset change value, and obtain a pixel detection area with the target pixel point as the center and the preset length as the radius;

[0137] The tracking image confirmation module is used to count the target ratio of target pixels and feature points in the pixel detection area. If the target ratio is greater than a preset ratio value, the detection image is recorded as a tracking image.

[0138] In one embodiment, the sentiment analysis module includes:

[0139] The activation signal generation module is used to bring the tracking image into the residual network to extract feature information, and activate the extracted feature information through Gumbel Softmax to obtain the activation signal;

[0140] A final fusion feature generation module is used to bring the activation signal and the tracking image into the working memory network and the long-term memory network respectively to obtain a first memory feature and a second memory feature, and obtain a final fusion feature according to the first memory feature and the second memory feature;

[0141] The emotion comparison module is used to bring the final fusion features into the preset database for emotion similarity comparison to obtain the emotion analysis results; the preset database is used to store the features corresponding to different emotions in different combinations.

[0142] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for detecting and responding to emotions based on a driver's seat, characterized in that: The method comprises: Collecting the driver's facial video data in real time, and performing frame extraction processing on the facial video data through a first preset time window to obtain a facial image set in a time series; Performing face region recognition on a target face image to obtain a target region image, and inputting the target region image into a spatial transformation network model to perform face alignment to obtain a detection image; the target face image is any one face image in the face image set; Bringing the detection image into a feature point detection model to perform feature point detection to obtain a facial feature point set, performing real-time facial tracking on the driver to obtain a tracking image based on the facial feature point set, and adjusting the first preset time window based on the tracking image in a time series; Perform sentiment analysis on the tracking images in time series and generate response plans based on the sentiment analysis results.

2. The method for detecting and coping with emotions based on a driver's seat according to claim 1, characterized in that: The target area image obtained by performing face area recognition on the target facial image includes: Performing data processing on the target facial image to obtain first data, and performing multi-feature extraction on the first data through central difference convolution to obtain multiple initial features; Multiple first features are obtained by performing spatial attention enhancement on each initial feature through multi-scale feature modulation; For each first feature, performing center difference feature extraction on the first feature through depth-wise separable center difference convolution to obtain a second feature; The target feature is obtained by fusing all the second features through multi-scale channel attention; A face region is identified through a region proposal network according to the target feature, and the target facial image is cut according to the face region to obtain a target region image.

3. The method for detecting and coping with emotions based on a driver's seat according to claim 2, characterized in that: Inputting the target area image into the spatial transformation network model to perform face alignment to obtain a detection image includes: Extracting affine transformation parameters of the target area image to obtain target affine parameters, and obtaining an affine transformation matrix according to the target affine parameters; Extracting the pixel value corresponding to each pixel in the target area image by bilinear interpolation; Performing pixel inverse mapping on the pixel value corresponding to each pixel according to the affine transformation matrix to obtain a mapping image; The target feature is mapped onto the mapping image, and convolution is performed through a convolution layer to obtain a detection image.

4. The method for detecting and coping with emotions based on a driver's seat according to claim 1, characterized in that: According to the facial feature point set, real-time facial tracking of the driver to obtain a tracking image comprises: Acquire the detection image in real time, obtain the feature points of the current image, and calculate the position change of each feature point between the current image and the previous detection image through Euclidean distance; If there is a pixel point whose position change is greater than the preset change value, the pixel point is recorded as the target pixel point, and the pixel detection area is obtained with the target pixel point as the center and the preset length as the radius; The target ratio of the target pixel points to the feature points in the pixel detection area is counted, and if the target ratio is greater than a preset ratio value, the detection image is recorded as a tracking image.

5. The method for detecting and coping with emotions based on a driver's seat according to claim 1, characterized in that: Sentiment analysis of tracking images over time series includes: Bringing the tracking image into the residual network to extract feature information, and activating the extracted feature information through GumbelSoftmax to obtain an activation signal; Bringing the activation signal and the tracking image into the working memory network and the long-term memory network respectively to obtain a first memory feature and a second memory feature, and obtaining a final fusion feature according to the first memory feature and the second memory feature; The final fusion feature is brought into a preset database for emotion similarity comparison to obtain the emotion analysis result; the preset database is used to store the features corresponding to different emotions in different combinations.

6. An emotion detection and response system based on the driver's seat, characterized in that: The system comprises: A facial image acquisition module, used for collecting the driver's facial video data in real time, and performing frame extraction processing on the facial video data through a first preset time window to obtain a facial image set in a time series; A face alignment module is used to perform face region recognition on a target face image to obtain a target region image, and input the target region image into a spatial transformation network model to perform face alignment to obtain a detection image; the target face image is any one of the face images in the face image set; A face tracking module, used for bringing the detection image into a feature point detection model to perform feature point detection to obtain a facial feature point set, performing real-time face tracking on the driver to obtain a tracking image based on the facial feature point set, and adjusting the first preset time window based on the tracking image in a time series; The sentiment analysis module is used to perform sentiment analysis on the tracking images in time series and generate response plans based on the sentiment analysis results.

7. The emotion detection and response system based on the driving position according to claim 6, characterized in that: The face alignment comprises: A multi-feature extraction module, used for performing data processing on the target facial image to obtain first data, and performing multi-feature extraction on the first data through central difference convolution to obtain multiple initial features; An attention enhancement module, configured to perform spatial attention enhancement on each initial feature through multi-scale feature modulation to obtain a plurality of first features; A center difference feature extraction module, used for performing center difference feature extraction on each first feature through a depth-separable center difference convolution to obtain a second feature; A feature fusion module is used to fuse all the second features to obtain the target feature through multi-scale channel attention; The face region recognition module is used to recognize the face region through a region proposal network according to the target features, and to cut the target facial image according to the face region to obtain a target region image.

8. The emotion detection and response system based on the driving position according to claim 7, characterized in that: The face alignment further includes: An affine parameter extraction module is used to extract affine transformation parameters from the target area image to obtain target affine parameters, and obtain an affine transformation matrix according to the target affine parameters; A pixel value extraction module, used to extract the pixel value corresponding to each pixel in the target area image through bilinear interpolation; A pixel mapping module, used for performing pixel inverse mapping on the pixel value corresponding to each pixel according to the affine transformation matrix to obtain a mapping image; The image convolution module is used to map the target features onto the mapping image and obtain a detection image by convolution through a convolution layer.

9. The emotion detection and response system based on the driving position according to claim 6, characterized in that: The face tracking module comprises: The position change calculation module is used to obtain the detection image in real time, obtain the feature points of the current image, and calculate the position change of each feature point between the current image and the previous detection image through the Euclidean distance; A pixel detection area determination module is used to record a pixel point as a target pixel point if the position change of a pixel point is greater than a preset change value, and obtain a pixel detection area with the target pixel point as the center and a preset length as the radius; The tracking image confirmation module is used to count the target ratio of target pixels and feature points in the pixel detection area. If the target ratio is greater than a preset ratio value, the detection image is recorded as a tracking image.

10. The emotion detection and response system based on the driving position according to claim 6, characterized in that: The sentiment analysis module includes: An activation signal generation module is used to bring the tracking image into the residual network to extract feature information, and activate the extracted feature information through Gumbel Softmax to obtain an activation signal; A final fusion feature generation module is used to bring the activation signal and the tracking image into the working memory network and the long-term memory network respectively to obtain a first memory feature and a second memory feature, and obtain a final fusion feature according to the first memory feature and the second memory feature; The emotion comparison module is used to bring the final fusion feature into a preset database for emotion similarity comparison to obtain emotion analysis results; the preset database is used to store features corresponding to different emotions in different combinations.

Images