Multimodal fusion emotional disorder detection device and method based on driving task

Abstract

The present invention discloses a multimodal fusion emotional disorder detection device and method based on driving tasks, in order to solve the problems of high labor cost, low detection efficiency and low accuracy of existing emotional disorder detection when performing emotional disorder detection on employees in special positions. Specifically, it includes a simulated cockpit, a data acquisition unit, a data processing unit and a power supply unit; the simulated cockpit is used to simulate a real driving environment for the person being tested; the data acquisition unit is used to collect the brain data, sound data, eye movement data and facial expression pictures of the person being tested; the data processing unit is respectively electrically connected to the microphone, infrared light group, infrared light camera, 3D depth laser camera, visible light camera and electrode probe contained in the data acquisition unit, and is used to process the brain data, sound data, eye movement data and facial expression pictures; the power supply unit is respectively electrically connected to each component of the data processing unit and the data processing unit.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotional disorder detection, and relates to a multi-modal fusion emotional disorder detection device and method based on driving tasks. Background Art

[0002] Depressive disorder, manic disorder and bipolar disorder are all mood disorders. In some special industries and special positions, people with mood disorders will not be able to perform their job duties, which will not only harm the company but also themselves. Therefore, it is necessary to test their emotions before taking up their posts. Among them: depressive disorder usually manifests "three lows", including: slow eye attention, slow thinking reaction and low emotional expression; manic disorder usually manifests "three highs", including: excited will and behavior, unrestrained and active thinking and high emotional expression; bipolar disorder usually manifests sometimes "three lows" and sometimes "three highs", or both at the same time.

[0003] For special positions, such emotional disorders need to be screened before taking up the post. Existing tests usually require the new employees to go to professional institutions for testing and be accompanied by professional testers. The testing method is to use DSM-V emotion scales to assess the emotional disorders of new employees. This method is time-consuming and labor-intensive, resulting in high labor costs and low testing efficiency. In addition, the test results are also subjective and random, without unified standards, and the testing process is single-mode, resulting in low accuracy of the test results. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal fusion emotion disorder detection device and method based on driving tasks, so as to solve the technical problems of high labor cost, low detection efficiency and low accuracy of existing emotion disorder detection when performing emotion disorder detection on employees in special positions.

[0005] In order to achieve the above-mentioned object, the present invention provides a multi-modal fusion emotional disorder detection device based on driving tasks, which is special in that it includes a simulated cockpit, a data acquisition unit, a data processing unit and a power supply unit;

[0006] The simulated cockpit is used to simulate a real driving environment for the person being tested;

[0007] The data acquisition unit includes a microphone, an infrared light group, an infrared camera, a 3D depth laser camera, a visible light camera and a plurality of electrode probes; the plurality of electrode probes are used to be distributed on the head of the subject to obtain the brain data of the subject; the microphone is used to obtain the sound data of the subject; the infrared light group is used to irradiate the infrared light spot to the eyeball of the subject; the infrared camera is used to capture the eye movement data of the pupil center of the subject relative to the infrared light spot in real time; the visible light camera is used to capture the facial photo of the subject; the 3D depth laser camera is used to extract the facial expression picture from the facial photo;

[0008] The data processing unit is electrically connected to the microphone, infrared light group, infrared light camera, 3D depth laser camera, visible light camera and electrode probe respectively, and is used to process the brain data, sound data, eye movement data and facial expression pictures;

[0009] The power supply unit is electrically connected to the microphone, infrared light group, infrared camera, 3D depth laser camera, visible light camera, electrode probe and data processing unit respectively, and is used to supply power to the data acquisition unit and the data processing unit.

[0010] Further, it also includes a storage unit;

[0011] The storage unit, the microphone, the infrared light group, the infrared camera, the 3D depth laser camera, the visible light camera, the electrode probe, and the power supply unit are electrically connected to store the sound data, the eye movement data, the brain data, and the facial expression pictures.

[0012] Furthermore, it also includes a helmet for the person being tested to wear;

[0013] A plurality of electrode probes are evenly distributed on the inner wall of the helmet;

[0014] The data processing unit and the power supply unit are installed on one side of the inner wall of the helmet;

[0015] An adjustment bracket is provided on the outer wall of the helmet;

[0016] One end of the adjustment bracket is connected to the helmet, and the other end points to the face of the person being tested;

[0017] The microphone, infrared light group, infrared light camera, 3D depth laser camera and visible light camera are all arranged at the other end of the adjustment bracket.

[0018] Furthermore, a circular mounting plate is provided at the other end of the adjustment bracket;

[0019] The infrared lamp group includes a plurality of infrared lamps arranged on the mounting plate, and the plurality of infrared lamps are distributed circumferentially on the mounting plate to form an infrared lamp circle; the microphone, infrared light camera, 3D depth laser camera and visible light camera are installed on the mounting plate and are located within the infrared lamp circle.

[0020] Further, the simulated cockpit includes a seat, a surround display screen, a steering wheel, a throttle and a brake;

[0021] The seat is used for the person being tested to sit on;

[0022] The wraparound display screen is arranged around the seat and is used to simulate and display the vehicle's windshield, the road conditions outside the windshield, the dashboard information inside the vehicle, and the rearview mirror information;

[0023] The steering wheel is arranged between the seat and the display screen, and is used for the person being tested to grasp;

[0024] The accelerator and the brake are arranged side by side below the display screen for the person being tested to step on.

[0025] The present invention also provides a multi-modal fusion emotional disorder detection method based on driving tasks, based on the above-mentioned multi-modal fusion emotional disorder detection device based on driving tasks, which is special in that it includes:

[0026] Step 1: Show the simulated road conditions to the subject in the cockpit, and make a preliminary assessment of the subject's emotional disorder using the following three methods:

[0027] Method 1: Collect the brain data of the subject through the electrode probe, and make a preliminary judgment on the emotional disorder of the subject based on the brain data;

[0028] Method 2: Collect the eye movement data of the subject through an infrared light group and an infrared camera, and make a preliminary judgment on the emotional disorder of the subject based on the eye movement data;

[0029] Method 3: Collect the voice data of the subject through a microphone, and collect the facial expression picture of the subject through a visible light camera and a 3D depth laser camera, and make a preliminary judgment on the emotional disorder of the subject based on the voice data and facial expression picture;

[0030] Step 2: Compare the three preliminary determination results of the three methods to determine the final determination result, wherein the preliminary determination results include normal mood, depressive mood disorder tendency and manic mood disorder tendency:

[0031] If the three preliminary judgment results are all normal emotions, then the final judgment result is determined to be normal emotions;

[0032] If at least one of the three preliminary determination results meets the criteria of mood disorder, and the preliminary determination results are consistent, the final determination result is determined to be a tendency toward manic mood disorder or a tendency toward depressive mood disorder;

[0033] If at least two of the three preliminary determination results meet the criteria of mood disorder, and the preliminary determination results are different, then the final determination result is determined to be bipolar mood disorder.

[0034] Furthermore, step 2 also includes determining the degree of emotional disorder:

[0035] If only one of the three preliminary determination results meets the criteria of emotional disorder, the degree of emotional disorder of the person being tested is judged to be mild;

[0036] If only two of the three preliminary determination results meet the criteria of emotional disorder, the degree of emotional disorder of the person being tested is determined to be moderate;

[0037] When all three preliminary determination results meet the criteria for emotional disorder, the degree of emotional disorder of the person being tested is determined to be severe.

[0038] Furthermore, in step 1, method 2 specifically includes:

[0039] B1. Irradiate the infrared light spots to the eyeballs of the person being tested through the infrared light group, and establish a coordinate system based on the infrared light spots;

[0040] B2, using an infrared camera to capture the discrete points of eye movement data of the pupil center of the subject in the coordinate system in real time;

[0041] B3, generating a Voronoi diagram according to the discrete points of the eye movement data;

[0042] B4, calculating the percentage of the area of ​​each polygonal region in the Voronoi diagram to the mean area of ​​all polygonal regions, and obtaining a normalized value of each polygonal region;

[0043] B5. A frequency distribution histogram is formed based on all normalized values, and a gamma distribution is used to fit the cumulative distribution to obtain the dispersion coefficient alpha;

[0044] B6. Make a preliminary assessment of the subject's emotional disorder based on the dispersion coefficient alpha;

[0045] When alpha < 0.2, the preliminary judgment result meets the criteria of depressive mood disorder;

[0046] When 0.2≤alpha≤0.5, the preliminary judgment result is that the mood is normal;

[0047] When alpha>0.5, the preliminary judgment result meets the criteria of manic disorder.

[0048] Further, in step 1, method 1 specifically includes:

[0049] A1. Collect the time delay information t and peak information h of the P300 evoked potential event response generated by the subject's brain through multiple electrode probes;

[0050] A2. Make a preliminary judgment on the tendency of the subject to emotional disorder based on the time delay information t and peak information h of the brain's P300 evoked potential event response;

[0051] When the time delay information t<150ms and h>1.5H, the preliminary judgment result meets the criteria of manic mood disorder;

[0052] When 150ms≤t≤350ms and 0.8H≤h≤1.5H, the preliminary judgment result is that the emotion is normal;

[0053] When t>350ms and h<0.8H, the preliminary judgment result meets the criteria for depressive mood disorder.

[0054] Further, in step 1, method three specifically includes:

[0055] C1. Establishing a mapping relationship between an emotional disorder dataset and an emotional disorder tendency, and building an extended convolutional neural network based on the mapping relationship; the emotional disorder dataset includes facial expression images and two-dimensional frequency domain information;

[0056] C2. Collect the sound data of the person being tested during the simulated driving process through a microphone, and collect the facial expression pictures of the person being tested through a visible light camera and a 3D depth laser camera;

[0057] C3, using discrete cosine transform to process the collected sound data and obtain a two-dimensional time-frequency diagram of time and frequency;

[0058] C4, matching the facial expression picture with the two-dimensional time-frequency graph based on the time relationship to obtain an emotional disorder data set of the subject;

[0059] C5, passing the emotional disorder data set of the subject obtained in step C4 into the extended convolutional neural network built in step C1 to determine the emotional disorder;

[0060] C6. Outputting a preliminary determination result of the emotional disorder of the subject; the emotional disorder includes normal mood, depressive mood disorder, and manic mood disorder.

[0061] Beneficial effects of the present invention:

[0062] 1. The multimodal fusion emotional disorder detection device based on driving tasks provided by the present invention is based on the particularity of driving tasks, that is, the attention line of sight needs to take into account the road information ahead and the dashboard information, and the brain thinking needs to operate the steering wheel and the accelerator and brake to respond to the vehicle conditions ahead in a timely manner, and at the same time test the emotional control expression of the person being tested when facing the driving task. Therefore, an electrode probe, a microphone, an infrared light group, an infrared camera, a 3D depth laser camera, and a visible light camera are set to collect brain data, sound data, eye movement data, and facial expression pictures of the person being tested during driving, so as to comprehensively judge the distribution of the attention mechanism, thinking reaction ability, and emotional control ability of the person being tested, without the need for human participation, thus reducing the cost of personnel screening in special industries and special positions, that is, reducing the cost of emotional disorder detection, and avoiding the uncertainty of human subjective judgment.

[0063] 2. The multimodal fusion emotion disorder detection device based on the driving task provided by the present invention can simultaneously collect the brain data, sound data, eye movement data and facial expression pictures of the person being tested, and obtain multiple research dimensions at the same time. Compared with traditional manual detection, it can integrate multiple functions to detect the multimodal data of the person being tested, without the need to repeat and frequently conduct experiments in different modalities, and can quickly and accurately obtain judgment results.

[0064] 3. The multimodal fusion emotional disorder detection device based on driving tasks provided by the present invention can be set as a unified integrated device to ensure consistent detection environment, consistent hardware equipment, consistent control variables, and consistent detection standards.

[0065] 4. The multimodal fusion mood disorder detection device based on driving tasks provided by the present invention can simultaneously detect and differentiate depressive mood disorders, manic mood disorders and bipolar mood disorders.

[0066] 5. The multimodal fusion emotional disorder detection device and method based on driving tasks provided by the present invention respectively examine the distribution of the attention mechanism, thinking reaction ability and emotional expression of the subject through eye tracking, brain data processing and emotion recognition technology, and make a comprehensive judgment to determine the emotional disorder tendency of the subject. This method relies on the multimodal fusion emotional disorder detection device provided by the present invention, reduces labor costs, speeds up and standardizes the detection process, improves detection efficiency, unifies the detection results, and improves the accuracy of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the structure of an embodiment of a multi-modal fusion emotional disorder detection device based on a driving task of the present invention;

[0068] Figure 2 is a schematic structural diagram of a helmet in an embodiment of the present invention;

[0069] Figure 3 is a flow chart of a multimodal fusion emotional disorder detection method based on driving tasks of the present invention;

[0070] Figure 4 is a schematic diagram of discrete points of eye movement data obtained in an embodiment of the present invention;

[0071] Figure 5 is a Voronoi diagram generated according to discrete points of eye movement data in an embodiment of the present invention;

[0072] Figure 6 It is the basic structure of the VGG16 neural network in the embodiment of the present invention;

[0073] Figure 7 Schematic diagram of an extended convolution algorithm in an embodiment of the present invention;

[0074] Among them, (a) is dilated = 1, (b) is dilated = 21, and (c) is dilated = 22, representing the setting of the dilation rate parameter. The white dots represent the parameters of the ordinary convolutional network, and the shaded area is the receptive field range. It can be seen that when the parameter quantity remains consistent, the receptive field range is greatly improved by expanding the convolution, so that the network training can obtain a higher classification accuracy without changing the parameter quantity.

[0075] Figure Number:

[0076] 1-data acquisition unit, 11-microphone, 12-infrared light group, 13-infrared camera, 14-3D depth laser camera, 15-visible light camera, 16-electrode probe, 2-data processing unit, 3-power supply unit, 4-storage unit, 5-helmet, 51-adjustment bracket, 61-seat, 62-display screen, 63-steering wheel, 64-accelerator, 65-brake. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Depressive disorder, manic disorder and bipolar disorder are all mood disorders. Bipolar disorder is different from depressive disorder and manic disorder. Bipolar disorder means that the person being tested has very significant and persistent emotional problems. In one stage, he or she will show a very obvious depressive state, and in other stages, he or she will show a manic state. These two extreme manifestations appear in the same person being tested. In addition to depressive disorder or manic disorder occurring alone, they may also appear mixed, that is, depressive disorder and manic disorder may occur at the same time. This usually occurs when depressive disorder and manic disorder quickly transition. The duration is generally shorter, and most people quickly transition to the manic phase or depressive phase.

[0079] Driving tasks are closely related to emotions. When faced with conditions such as traffic congestion, bad weather, and other drivers' aggressive driving behavior, the driver's manic disorder is easily stimulated. In addition, driving tasks can also be affected by depressive disorders, and vehicle accidents caused by inability to concentrate and slow thinking reactions. Therefore, driving tasks can be a good test of an individual's tendency to emotional disorders.

[0080] During driving, the driver's attention needs to be focused on both the road ahead and the dashboard, and the driver's brain needs to operate the steering wheel and the accelerator and brake to respond to the vehicle ahead in a timely manner. This also tests the driver's emotional control when facing driving tasks. Therefore, driving tasks can comprehensively detect the distribution of an individual's attention mechanism, thinking reaction ability, and emotional control ability, which correspond to depressive disorder, manic disorder, and bipolar disorder, respectively.

[0081] To this end, an embodiment of the present invention provides a multi-modal fusion emotional disorder detection device based on driving tasks, which may include a simulated cockpit, a data acquisition unit 1, a data processing unit 2, a power supply unit 3, and a storage unit 4;

[0082] Figure 1 is a schematic diagram of the structure of an embodiment of a multi-modal fusion emotional disorder detection device based on a driving task of the present invention, such as Figure 1 As shown, the simulated cockpit is used to simulate a real driving environment for the person being tested; the simulated cockpit includes a seat 61, a surround display screen 62, a steering wheel 63, an accelerator 64 and a brake 65; the seat 61 is for the person being tested to sit on; the surround display screen 62 is arranged around the seat 61, and is used to simulate and display the vehicle's windshield, the road conditions outside the windshield, the dashboard information in the vehicle, and the rearview mirror information, etc.; the steering wheel 63 is arranged between the seat 61 and the display screen 62, for the person being tested to grasp; the accelerator 64 and the brake 65 are arranged side by side below the display screen 62, for the person being tested to step on.

[0083] The data acquisition unit 1 includes a microphone 11, an infrared light group 12, an infrared camera 13, a 3D depth laser camera 14, a visible light camera 15 and 128 electrode probes 16; in addition, it also includes a helmet 5 for the person being tested to wear. Figure 2 : is a schematic diagram of the structure of a helmet in an embodiment of the present invention, such as Figure 2 As shown, 128 electrode probes 16 are evenly embedded on the inner wall of the helmet 5 to cover various areas of the subject's brain, so as to obtain the subject's brain data, including the overall brain wave data during the acquisition and detection of event-related potential physiological signals of the activation state of the cerebral cortex; an adjustment bracket 51 is provided on the outer wall of the helmet 5; one end of the adjustment bracket 51 is connected to the helmet 5, and the other end points to the face of the subject; a circular mounting plate is provided at the other end of the adjustment bracket 51; the adjustment bracket 51 can freely adjust the height, angle and distance of the mounting plate and the components above it to adapt to different subjects; the infrared lamp group 12 includes a plurality of infrared lamps arranged on the mounting plate, and the plurality of infrared lamps are distributed circumferentially on the mounting plate to form an infrared lamp circle; the microphone 11, the infrared light camera 13, the 3D depth laser camera 14 and the visible light camera 15 are installed on the mounting plate and are located in the infrared lamp circle. The infrared light group 12 is used to irradiate infrared light spots to the eyeballs of the person being tested to divide the pupil and iris; the microphone 11 is used to obtain the sound data of the person being tested; the infrared light camera 13 is used to obtain the coordinates of the center of the pupil, and use the infrared light spot on the eyeball to establish a coordinate mapping relationship with the center of the pupil, so as to capture the eye movement data of the center of the pupil of the person being tested relative to the infrared light spot in real time; the visible light camera 15 is used to capture the facial photo of the person being tested; the 3D depth laser camera 14 is used to extract facial expression pictures from the facial photos, and finally use the visual algorithm to detect and identify the emotions of the person being tested in real time. It is worth noting that the microphone 11, the infrared light group 12, the infrared light camera 13, the 3D depth laser camera 14 and the visible light camera 15 can also be centrally arranged in front of the display screen 62 and above the steering wheel. The collection of brain data, sound data, eye movement data and facial expression pictures is carried out simultaneously.

[0084] The data processing unit 2, the power supply unit 3 and the storage unit 4 are installed on one side of the inner wall of the helmet 5; the data processing unit 2 is electrically connected to the microphone 11, the infrared light group 12, the infrared light camera 13, the 3D depth laser camera 14, the visible light camera 15 and the electrode probe 16, respectively, for processing brain data, sound data, eye movement data and facial expression pictures; the data processing unit 2 provides a data input unit, a computing unit and an output unit. The data input unit will receive the brain data, sound data, eye movement data and facial expression pictures of the subject in real time; the computing unit is responsible for calling the algorithm model to process the brain data, eye movement data and facial expression pictures in real time; the output unit is used to give the result of the determination of the tendency of emotional disorder, including whether the subject has emotional disorder, the tendency of emotional disorder and the severity of emotional disorder, so as to judge whether the subject is suitable for a special industry or a special position.

[0085] The power supply unit 3 is electrically connected to the microphone 11, the infrared light group 12, the infrared light camera 13, the 3D depth laser camera 14, the visible light camera 15, the electrode probe 16, the data processing unit 2 and the storage unit 4, respectively, and is used to supply real-time power to the microphone 11, the infrared light group 12, the infrared light camera 13, the 3D depth laser camera 14, the visible light camera 15, the electrode probe 16, the data processing unit 2 and the storage unit 4. The storage unit 4 is used to store the brain data, sound data, eye movement data and facial expression pictures collected by the data collection unit 1.

[0086] The multimodal fusion emotional disorder detection device based on driving tasks has built a multimodal driving cockpit hardware platform, integrating eye tracking technology, EEG signal processing technology and 3D emotion recognition technology. It has the characteristics of fast and standardized process, high detection accuracy and wide applicability, which can greatly reduce the labor cost of emotional disorder detection. The surround display screen displays the first-person perspective of the interior of the car in real time and provides a real-time driving environment. The hardware devices for driving operations (such as accelerator and steering wheel) can be synchronously connected through software adaptation. The driving environment sets the traffic rules of intelligent transportation systems, unpredictable pedestrians and sudden dangerous situations. The subjects will sit in the driving seat, operate the steering wheel, accelerator and brake, drive in this driving environment, and also receive emotional disorder detection. The detection process does not require human participation, making the detection results objective and accurate, saving labor costs.

[0087] Based on the above-mentioned multimodal fusion emotional disorder detection device, the embodiment of the present invention also provides a multimodal fusion emotional disorder detection method based on driving tasks, such as Figure 3 As shown, the following steps may be included:

[0088] Step 1: Show the simulated road conditions to the subject in the cockpit. The subject sits on the seat, wears a helmet, and calibrates the microphone 11, infrared light group 12, infrared light camera 13, 3D depth laser camera 14, visible light camera 15 and electrode probe 16. While the subject is driving, the subject's tendency to emotional disorder is preliminarily determined by the following three methods:

[0089] Method 1: Collect brain data of the subject through the electrode probe 16, and make a preliminary judgment on the subject's tendency of emotional disorder based on the brain data; specifically including:

[0090] A1. For the unpredictable appearance of pedestrians and traffic accidents in sudden dangerous situations during the driving task, such as the sudden lane change or braking of the vehicle in front, extract the EEG data fragments during the entire driving event, first re-reference and filter (10-80Hz): Event-related brain data is concentrated in 10-80Hz, including beta band brain waves and gamma band brain waves with significant differences. Beta band brain waves: The frequency is usually 10 to 30Hz, usually associated with active, busy or anxious thinking states, and is also affected by attention. It is used to analyze the degree of excitement of the human body at a certain moment. Gamma band brain waves: The frequency is above 30Hz, which is the fastest brain wave and is related to large-scale brain network activities and cognitive phenomena, such as working memory, attention, etc.

[0091] Using the visual evoked potential in the ERP event-related potential, the time delay information t and peak information h of the P300 evoked potential event response generated by the subject's brain are collected through multiple electrode probes 16 as an indicator for measuring the subject's reaction speed; the ERP event-related potential reflects the changes in the brain's neuroelectrophysiology during the cognitive process, also known as cognitive potential, which refers to the brain potential recorded from the surface of the skull when people perform cognitive processing on a certain topic. This embodiment can use visual evoked potential (VEP). P300 represents a positive wave that should appear in 300ms, and P300 is generally believed to be related to human cognitive function. If the neural pathway is abnormal, the latency and peak value of the characteristic wave will change, generally manifested as an extension of the latency and a weakening of the peak value.

[0092] A2. Make a preliminary judgment on the tendency of the subject to emotional disorder based on the time delay information t and peak information h of the brain's P300 evoked potential event response;

[0093] When the time delay information t<150ms and h>1.5H (H is the unit of intensity), the preliminary judgment result meets the criteria of manic mood disorder;

[0094] When 150ms≤t≤350ms and 0.8H≤h≤1.5H, the preliminary judgment result is that the emotion is normal;

[0095] When t>350ms and h<0.8H, the preliminary judgment result meets the criteria for depressive mood disorder.

[0096] Method 2: collecting the eye movement data of the subject during the entire driving event through the infrared light group 12 and the infrared light camera 13, and making a preliminary judgment on the subject's tendency of emotional disorder based on the eye movement data; specifically including:

[0097] B1. When the subject is driving, infrared light spots are irradiated to the eyeballs of the subject through the infrared light group 12, and a coordinate system is established based on the infrared light spots;

[0098] B2, such as Figure 4 As shown, the infrared camera 13 is used to capture the discrete points of the eye movement data of the pupil center of the subject in the coordinate system in real time;

[0099] B3, such as Figure 5 As shown, a Voronoi diagram is generated based on the discrete points of eye movement data, that is, a continuous polygon composed of a set of perpendicular bisectors connecting the straight lines of two adjacent points;

[0100] B4. Calculate the percentage of the area of ​​each polygonal region in the Voronoi diagram to the mean area of ​​all polygonal regions, and obtain the normalized value of each polygonal region;

[0101] B5. A frequency distribution histogram is formed based on all normalized values. The gamma distribution is used to fit the cumulative distribution to obtain the dispersion coefficient alpha, where alpha is a shape parameter that can be used to measure the degree of aggregation of discrete data points. The larger the alpha value, the more dispersed the distribution.

[0102] B6. Make a preliminary assessment of the subject's tendency toward emotional disorder based on the dispersion coefficient alpha;

[0103] When alpha < 0.2, the preliminary judgment result meets the criteria of depressive mood disorder;

[0104] When 0.2≤alpha≤0.5, the preliminary judgment result is that the mood is normal;

[0105] When alpha>0.5, the preliminary judgment result meets the criteria of manic disorder.

[0106] Method 3: The microphone 11 is used to collect the voice data of the subject to be tested, and the visible light camera 15 and the 3D depth laser camera 14 are used to collect the facial expression pictures of the subject to be tested. Based on the voice data and the facial expression pictures, i.e., the 3D emotion recognition data, a preliminary judgment is made on the tendency of the subject to be emotionally disturbed; specifically, the following are included:

[0107] C1. Establish a mapping relationship between the emotional disorder dataset and the emotional disorder tendency, and build an extended convolutional neural network based on the mapping relationship; the emotional disorder dataset includes facial expression images and two-dimensional frequency domain information;

[0108] C2. The microphone 11 is used to collect the sound data of the person being tested during the simulated driving process, and the facial data is obtained through the visible light camera. Then the 3D depth laser camera 14 projects the laser. When it encounters the face of the person, the laser is reflected back to the camera, and the facial expression picture is cropped. The 3D depth laser camera 14 can extract the depth features that RGB visible light cannot extract, and fuse and extract micro-expressions.

[0109] C3. Process the collected sound data (including voice, tone and other voiceprint data) using discrete cosine transform to obtain a two-dimensional time-frequency diagram of time and frequency;

[0110] C4, matching the facial expression image with the two-dimensional time-frequency graph based on the time relationship to obtain the emotional disorder dataset of the subject;

[0111] C5, the emotional disorder data set of the subject obtained in step C4 is passed to the extended convolutional neural network built in step C1 to determine the emotional disorder; combined with Figure 6 and Figure 7 As shown in the figure, the extended convolution neural network is based on the VGG16 neural network. By increasing the depth, it can effectively improve the performance. At the same time, convolution can replace full connection and can adapt to pictures of various sizes. The convolution layer uses dilated convolution. Dilated is a mode of kernel sparsification, and stride is just a special case of dilated. Compared with ordinary convolution, in addition to the same size of the convolution kernel, the extended convolution also has a dilation rate parameter, which is mainly used to indicate the size of the dilation. In the case of the same number of parameters in the neural network, that is, the size of the output feature map remains unchanged, the extended convolution has a larger receptive field. Here, the dilation rate parameter is set to four times. Similarly, when the receptive field is the same, the number of parameters can be greatly reduced compared to ordinary convolution. For example, a 3×3 convolution kernel with a dilation rate of 2 has the same receptive field as a 5×5 convolution kernel, but the number of parameters is only 9, which is about one-third of the number of parameters of a 5×5 convolution.

[0112] Using the pooling layer as the boundary, VGG16 has a total of 6 block structures, and the number of channels in each block structure is the same. Because the convolutional layer and the fully connected layer have weight coefficients, they are also called weight layers. There are 13 convolutional layers, 3 fully connected layers, and the pooling layer does not involve weights. So there are 13+3=16 layers in total.

[0113] For the VGG16 convolutional neural network, its 13 convolutional layers and 5 pooling layers are responsible for feature extraction, and the last 3 fully connected layers are responsible for completing the classification task. Based on the extended convolutional neural network, the voiceprint module of deep facial features, voice and tone is integrated to perform deep learning to broaden the convolutional neural network discrimination, and finally give a preliminary judgment result.

[0114] C6. Outputting a preliminary determination result of the emotional disorder of the subject; the emotional disorder includes normal mood, depressive mood disorder, and manic mood disorder.

[0115] Step 2: Compare the three preliminary judgment results of the three methods to determine the final judgment result. The preliminary judgment results include normal mood, depressive mood disorder tendency and manic mood disorder tendency:

[0116] If the three preliminary judgment results are all normal emotions, then the final judgment result is determined to be normal emotions;

[0117] If at least one of the three preliminary determination results meets the requirement of mood disorder tendency, and the preliminary determination results are consistent, the final determination result is determined to be manic mood disorder tendency or depressive mood disorder tendency;

[0118] If at least two of the three preliminary determination results meet the tendency of mood disorder, and the preliminary determination results are different, the final determination result is determined to be bipolar mood disorder.

[0119] In addition, the determination of the degree of tendency to emotional disorder is also included before or after step 2:

[0120] If only one of the three preliminary determination results meets the requirement of emotional disorder tendency, the degree of emotional disorder tendency of the tested person is judged to be mild;

[0121] If only two of the three preliminary determination results meet the emotional disorder tendency, the degree of emotional disorder tendency of the tested person is determined to be moderate;

[0122] When all three preliminary determination results satisfy the tendency toward emotional disorder, the degree of emotional disorder tendency of the person being tested is determined to be severe.

[0123] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0124] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal fusion emotional disorder detection method based on driving tasks, characterized in that: A multimodal fusion emotional disorder detection device is used, the detection device comprising a simulated cockpit, a data acquisition unit (1), a data processing unit (2) and a power supply unit (3); The simulated cockpit is used to simulate a real driving environment for the person being tested; The data acquisition unit (1) comprises a microphone (11), an infrared light group (12), an infrared camera (13), a 3D depth laser camera (14), a visible light camera (15) and a plurality of electrode probes (16); the plurality of electrode probes (16) are used to be distributed on the head of the subject to obtain the brain data of the subject; the microphone (11) is used to obtain the sound data of the subject; the infrared light group (12) is used to irradiate the infrared light spot to the eyeball of the subject; the infrared camera (13) is used to capture the eye movement data of the pupil center of the subject relative to the infrared light spot in real time; the visible light camera (15) is used to capture the facial photo of the subject; and the 3D depth laser camera (14) is used to extract the facial expression picture from the facial photo; The data processing unit (2) is electrically connected to the microphone (11), the infrared light group (12), the infrared light camera (13), the 3D depth laser camera (14), the visible light camera (15) and the electrode probe (16) respectively, and is used to process the brain data, sound data, eye movement data and facial expression pictures; The power supply unit (3) is electrically connected to the microphone (11), the infrared light group (12), the infrared light camera (13), the 3D depth laser camera (14), the visible light camera (15), the electrode probe (16) and the data processing unit (2) respectively, and is used to supply power to the data acquisition unit (1) and the data processing unit (2); The detection method comprises the following steps: Step 1: Show the simulated road conditions to the subject in the cockpit, and make a preliminary assessment of the subject's emotional disorder using the following three methods: Method 1: collecting brain data of the subject through the electrode probe (16), and making a preliminary assessment of the subject's emotional disorder based on the brain data; Method 2: collecting the eye movement data of the subject through the infrared light group (12) and the infrared light camera (13), and making a preliminary judgment on the emotional disorder of the subject based on the eye movement data; specifically including: B1, irradiating infrared light spots onto the eyeballs of the subject through the infrared light group (12), and establishing a coordinate system based on the infrared light spots; B2, capturing the eye movement data discrete points of the pupil center of the subject in the coordinate system in real time through an infrared camera (13); B3, generating a Voronoi diagram according to the discrete points of the eye movement data; B4, calculating the percentage of the area of ​​each polygonal region in the Voronoi diagram to the mean area of ​​all polygonal regions, and obtaining a normalized value of each polygonal region; B5. A frequency distribution histogram is formed based on all normalized values, and a gamma distribution is used to fit the cumulative distribution to obtain the dispersion coefficient alpha; B6. Make a preliminary assessment of the subject's emotional disorder based on the dispersion coefficient alpha; When alpha < 0.2, the preliminary judgment result meets the criteria of depressive mood disorder; When 0.2≤alpha≤0.5, the preliminary judgment result is that the mood is normal; When alpha>0.5, the preliminary judgment result meets the criteria of manic mood disorder; Method three: collecting voice data of the subject to be tested through a microphone (11), and collecting facial expression pictures of the subject to be tested through a visible light camera (15) and a 3D depth laser camera (14), and making a preliminary judgment on the emotional disorder of the subject to be tested based on the voice data and the facial expression pictures; Step 2: Compare the three preliminary determination results of the three methods to determine the final determination result, wherein the preliminary determination results include normal mood, depressive mood disorder tendency and manic mood disorder tendency: If the three preliminary judgment results are all normal emotions, then the final judgment result is determined to be normal emotions; If at least one of the three preliminary determination results meets the criteria of mood disorder, and the preliminary determination results are consistent, the final determination result is determined to be a tendency toward manic mood disorder or a tendency toward depressive mood disorder; If at least two of the three preliminary determination results meet the criteria of mood disorder, and the preliminary determination results are different, then the final determination result is determined to be bipolar mood disorder.

2. The multimodal fusion emotional disorder detection method based on driving task according to claim 1 is characterized in that: Step 2 also includes a determination of the degree of emotional disturbance: If only one of the three preliminary determination results meets the criteria of emotional disorder, the degree of emotional disorder of the person being tested is judged to be mild; If only two of the three preliminary determination results meet the criteria of emotional disorder, the degree of emotional disorder of the person being tested is determined to be moderate; When all three preliminary determination results meet the criteria for emotional disorder, the degree of emotional disorder of the person being tested is determined to be severe.

3. The multimodal fusion emotional disorder detection method based on driving task according to claim 2 is characterized in that: In step 1, method 1 specifically includes: A1, collecting the time delay information t and peak information h of the P300 evoked potential event response generated by the brain of the subject through multiple electrode probes (16); A2. Make a preliminary judgment on the tendency of the subject to emotional disorder based on the time delay information t and peak information h of the brain's P300 evoked potential event response; When the time delay information t<150ms and h>1.5H, the preliminary judgment result meets the criteria of manic mood disorder; When 150ms≤t≤350ms and 0.8H≤h≤1.5H, the preliminary judgment result is that the emotion is normal; When t>350ms and h<0.8H, the preliminary judgment result meets the criteria for depressive mood disorder.

4. The multimodal fusion emotional disorder detection method based on driving task according to claim 3 is characterized in that: In step 1, method 3 specifically includes: C1. Establishing a mapping relationship between an emotional disorder dataset and an emotional disorder tendency, and building an extended convolutional neural network based on the mapping relationship; the emotional disorder dataset includes facial expression images and two-dimensional frequency domain information; C2, collecting the sound data of the person being tested during the simulated driving process through the microphone (11), and collecting the facial expression picture of the person being tested through the visible light camera (15) and the 3D depth laser camera (14); C3, using discrete cosine transform to process the collected sound data and obtain a two-dimensional time-frequency diagram of time and frequency; C4, matching the facial expression picture with the two-dimensional time-frequency graph based on the time relationship to obtain an emotional disorder data set of the subject; C5, passing the emotional disorder data set of the subject obtained in step C4 into the extended convolutional neural network built in step C1 to determine the emotional disorder; C6. Outputting a preliminary determination result of the emotional disorder of the subject; the emotional disorder includes normal mood, depressive mood disorder, and manic mood disorder.

5. The multimodal fusion emotional disorder detection method based on driving task according to claim 1 is characterized by: The detection device also includes a storage unit (4); The storage unit (4) is electrically connected to the microphone (11), the infrared light group (12), the infrared light camera (13), the 3D depth laser camera (14), the visible light camera (15), the electrode probe (16) and the power supply unit (3), and is used to store the sound data, eye movement data, brain data and facial expression pictures.

6. The multimodal fusion emotional disorder detection method based on driving task according to claim 5 is characterized by: The detection device also includes a helmet (5) for the person being detected to wear; The plurality of electrode probes (16) are evenly distributed on the inner wall of the helmet (5); The data processing unit (2) and the power supply unit (3) are mounted on one side of the inner wall of the helmet (5); An adjustment bracket (51) is provided on the outer wall of the helmet (5); One end of the adjustment bracket (51) is connected to the helmet (5), and the other end points to the face of the person being tested; The microphone (11), the infrared light group (12), the infrared light camera (13), the 3D depth laser camera (14) and the visible light camera (15) are all arranged at the other end of the adjustment bracket (51).

7. The multimodal fusion emotional disorder detection method based on driving task according to claim 6 is characterized by: The other end of the adjustment bracket (51) is provided with a circular mounting plate; The infrared lamp group (12) includes a plurality of infrared lamps arranged on the mounting plate, and the plurality of infrared lamps are distributed circumferentially on the mounting plate to form an infrared lamp circle; the microphone (11), infrared light camera (13), 3D depth laser camera (14) and visible light camera (15) are installed on the mounting plate and are located within the infrared lamp circle.

8. The multimodal fusion emotional disorder detection method based on driving task according to claim 7 is characterized by: The simulated cockpit includes a seat (61), a surround display screen (62), a steering wheel (63), a throttle (64) and a brake (65); The seat (61) is used for the person being tested to sit on; The surround display screen (62) is arranged around the seat (61) and is used to simulate and display the vehicle's windshield, the road conditions outside the windshield, the dashboard information inside the vehicle, and the rearview mirror information; The steering wheel (63) is arranged between the seat (61) and the display screen (62) and is used for the person being tested to grasp; The accelerator (64) and the brake (65) are arranged side by side below the display screen (62) for the subject to step on.

Images