Multimodal student sleepiness monitoring method based on evidence theory

Abstract

The present invention discloses a multimodal student drowsiness monitoring method based on evidence theory. The drowsiness monitoring system as a whole uses visual acquisition equipment, electroencephalogram (EEG) acquisition equipment, and heart rate acquisition equipment to collect posture data, EEG data, and electrocardiogram (ECG) signal data generated by students during online learning. Based on the improved D-S evidence theory, these multimodal data are fused and calculated to improve monitoring accuracy, improve student learning efficiency, and assist teachers in their teaching work. An interface is used to open a data transmission channel between the teaching terminal and the drowsiness monitoring service system. Feedback on student drowsiness is provided to teachers through the teaching terminal, and sleepy students are reminded to pay attention and listen to the lecture. The present invention solves the paradoxical situation that violates common sense when traditional D-S evidence theory fuses high-conflict data. The advantage of the method proposed by the present invention is that even if high data conflict occurs during the fusion process, the fused result does not violate common sense.

Description

Technical Field

[0001] The present invention relates to the fields of multimodal data fusion, artificial intelligence, remote teaching, and in particular to a multimodal student sleepiness monitoring method based on evidence theory. Background Art

[0002] In today's society, online education and teaching have become the norm, gaining acceptance among teachers and students. Distance learning offers advantages such as convenience and low cost. However, with the widespread adoption of distance learning, while teachers can easily monitor students' performance and provide accurate reminders in traditional classroom settings, online learning faces limitations. Students are more susceptible to fatigue or drowsiness during online lectures, making it difficult for teachers to accurately assess their performance from a multitude of video frames. This negative impact on learning efficiency is significant, leading to the need for a drowsiness monitoring system and method to address this issue. Research using single-modal data to analyze student learning performance has already emerged, both domestically and internationally. However, the results obtained from combining multiple modalities are more convincing than from a single source, leading to the emergence of research based on multimodal data fusion. Using artificial intelligence algorithms to analyze image data generated during online learning is currently the mainstream approach. However, current online learning research, both domestically and internationally, also incorporates EEG and heart rate data. The integration of these diverse modalities to monitor and provide reminders for students' drowsiness during online learning is a worthy research question. DS evidence theory is a good method for decision-level fusion in data fusion. However, DS evidence theory will encounter paradoxical problems that violate common sense when fusing conflicting evidence. In the process of detecting students' drowsiness, data conflicts exist in the collected multi-source data due to deployment limitations of acquisition equipment, process defects, data missing, etc. Therefore, the present invention first improves the DS evidence theory, and then invents a multimodal student online learning drowsiness monitoring system and method based on the improved DS evidence theory. Summary of the Invention

[0003] The purpose of the present invention is to provide a multimodal student drowsiness monitoring method based on evidence theory, so as to solve the above-mentioned problems. The drowsiness monitoring system as a whole adopts visual acquisition equipment, EEG acquisition equipment, and heart rate acquisition equipment to collect the posture data, brain wave data, and ECG signal data generated by students when they are learning online. Based on the improved DS evidence theory, these multimodal data are fused and calculated to improve the monitoring accuracy, improve the students' learning efficiency and assist teachers in teaching. The data transmission channel between the teaching terminal and the drowsiness monitoring service system is opened by means of an interface, and the teacher is fed back the students' drowsiness through the teaching terminal and is reminded to pay attention to the lecture.

[0004] In order to achieve the above-mentioned object, the technical solution provided by the present invention is: a multimodal student sleepiness monitoring method based on evidence theory, comprising the following steps:

[0005] S1: After the teacher starts the class, the teaching terminal sends a signal to the sleepiness monitoring service system. The system activates the visual acquisition device, EEG device, and heart rate device for each student to start collecting the student's online learning data.

[0006] S2: Input the student images collected by the visual acquisition device into the trained facial feature point extraction model according to the preset time interval, and calculate the eye closing frequency data, sleepy nodding frequency data, and yawning frequency data based on the obtained feature point coordinates around the eyes, nose, and mouth; calculate the student's EEG data collected by the EEG device to obtain the ratio of the relative energy of the brain waves within the time interval; calculate the student's ECG signal data collected by the heart rate device to obtain the student's heart rate data within the time interval;

[0007] S3, based on the five types of data obtained, diagnose the local sleepiness state of the single-modal data. The sleepiness state is divided into three categories: A sleepy, B awake, and C unknown. The basic probability function of the three categories of states is assigned to the single-modal data, and five pieces of evidence are obtained;

[0008] S4, using DS evidence theory, integrates the evidence provided by the five types of data into decision-making;

[0009] S5, determining whether the fused student sleepiness value is greater than a threshold value, if so, the student is deemed to be in a sleepy state and executing S6, otherwise the student is deemed to be awake and executing S1;

[0010] S6, the teacher is informed of the student's sleepiness through the teaching terminal, and the student is reminded to pay attention and listen to the lecture. The system records the monitoring data;

[0011] S7, continuous monitoring, to determine whether the course has ended, if not, continue to execute S1, if the course has ended, shut down the system and perform data statistics.

[0012] Furthermore, the specific implementation method of obtaining the trained facial feature point extraction model in S2 is as follows;

[0013] S21, build a face sample library for students when learning, including face images of people with head-mounted EEG devices and public face libraries;

[0014] S22, preprocessing the collected sample images by data enhancement and labeling;

[0015] S23 uses the YOLO-V5 neural network to train and test in the face sample library, and continuously optimizes it through parameter adjustment and different image processing methods to obtain a high-precision facial feature extraction model.

[0016] Furthermore, the specific implementation method of obtaining the eye closing frequency data, sleepy nodding frequency data, and yawning frequency data in S2 is as follows;

[0017] (1) Eye closure frequency data: The average value of the aspect ratio EAR of the left and right eyes, EAR_avg, is calculated by the coordinates of the six feature points around the eyes, as shown in formula (1). When EAR_avg is less than the threshold of 0.2, it is judged as eye closure;

[0018]

[0019] Where p1-p6 represent the six coordinates of the eye feature points, ||p i -p j || means p i to p j The Euclidean distance, EAR_l represents the aspect ratio of the left eye, and EAR_r represents the aspect ratio of the right eye;

[0020] The PERCLOS algorithm is used to calculate the degree of eye closure of a person over a period of time. When a person is in a fatigue state, the number of eye closure frames increases significantly, and the PERCLOS parameter value increases accordingly. The calculation of the PERCLOS value is shown in formula (2). When its value is within the predetermined threshold range, according to P 80 Standard, that is, when the PERCLOS value is greater than or equal to 80%, it is determined to be a drowsy state;

[0021]

[0022] Among them, n is the total number of video frames within the detection time, i is the current frame number; Blink[i] is the number of closed-eye frames, is the total number of frames with closed eyes within the detection time;

[0023] (2) Sleepy nodding frequency data: The feature points around the nose are calculated to determine whether the change amplitude of the nose along the vertical axis in two consecutive frames is greater than the threshold. If the nodding duration is 3 seconds, it is determined to be sleepy nodding and the patient is in a sleepy state.

[0024] (3) Yawning frequency data: The mouth aspect ratio (MAR) is based on the ratio of the distance between the feature points in the horizontal and vertical directions of the human mouth. Formula (3) is used to determine whether the test subject is opening his mouth and yawning. According to experience, when MAR is greater than or equal to 0.75, it is determined to be yawning and sleepy;

[0025]

[0026] Where p1-p6 represent the coordinates of the six feature points around the mouth.

[0027] Furthermore, the specific implementation of step S4 is as follows: the brainwave relative energy ratio data is calculated using the following formula:

[0028]

[0029] Among them, r is the spectral power distribution, α, θ, and β represent the frequencies of α waves, θ waves, and β waves, respectively. According to experience, when the r value is less than 1.46, it is judged to be a drowsy state.

[0030] Furthermore, the frequency of the ECG signal within 1 minute, i.e., the heart rate data, is calculated using the following formula:

[0031]

[0032] Where X is the harmonic amplitude, F is the number of FFT sampling points, and f s is the FFT sampling frequency. When the f value is between 55 and 65, it is determined to be a drowsy state.

[0033] Furthermore, the basic probability function allocation refers to calculating the probability value of the student's sleepy state in each data through the proportion of sleepy states detected within a period of time, and defining the judgment of errors or missing data or unclear states as unclear states C; the evidence provided by the eye-closing frequency data is expressed as Z1: m1(A)=a1, m1(B)=a2, m1(C=a3, which means that the probability of the student being judged as sleepy by the eye-closing frequency data during this period is a1, the probability of being awake is a2, and the probability of being unclear is a3. The evidence provided by the nodding frequency data is represented by Z2: m2; the evidence provided by the yawning frequency data is represented by Z3: m3; the evidence provided by the brain wave relative energy ratio data is represented by Z4: m4; the evidence provided by the heart rate data is represented by Z5: m5; using the DS evidence theory, the evidence provided by the five types of data are fused for decision-making to obtain the fused sleepiness level result m 1,2,3,4,5 :m(A)=a, m(B)=b, m(C)=c.

[0034] Furthermore, the specific implementation of DS evidence theory in step S4 is as follows;

[0035] S41, using the Jousellme distance, calculates the distance between each pair of evidence, and calculates the similarity between each pair of evidence based on the distance, and then constructs the adjacency matrix of the evidence relationship;

[0036] S42, calculate the modification weight based on the direct and indirect relationships between the evidence. Here, the relationship strength of evidence is defined as the sum of the similarities between a certain evidence body and other evidence bodies. The relationship strength calculation formula is as shown in formula (8);

[0037]

[0038] In the formula, i and j represent the number of evidence, N i is the set of evidence numbers other than evidence i, Sim is the similarity function;

[0039] The modification weight of the direct relationship between evidences is calculated using the relationship strength, as shown in formula (9);

[0040]

[0041] In the formula, n represents the total number of evidences, and here n is 5. represents the sum of the relationship strengths of the five pieces of evidence;

[0042] S43, by removing a specific body of evidence to form a new set of evidence, the difference between the relationship strength in the original evidence set and the relationship strength in the new evidence set reflects the indirect relationship. The relationship coefficient of the indirect relationship of evidence is calculated as shown in formula (10);

[0043]

[0044] Where VS j is to remove m i Front m j Relationship strength, VS j ′ is to remove m i After m j The relationship strength of the indirect relationship is calculated by the indirect relationship coefficient λ of the evidence, as shown in formula (11);

[0045]

[0046] S44, integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight, as shown in formula (12),

[0047]

[0048] S45, finally, the original evidence is weighted and modified by weighted average of evidence, as shown in formula (13);

[0049]

[0050] S46, after modifying the evidence body, the evidence is then combined using the DS combination rule to obtain the final fusion result.

[0051] Furthermore, the distance calculation formula is as follows;

[0052]

[0053] in, Represents the vector form of evidence i and evidence j. D is an N×N matrix, where N is the number of elements in the union of evidence i and evidence j. The elements in D are defined as: |·| represents the cardinality of set ·, that is, the number of elements in set ·, the ratio of the intersection and union of A and B;

[0054] The similarity calculation formula is as follows:

[0055] Sim(m i ,m j )=1-d ij (m i ,m j ) (7).

[0056] Furthermore, the sleepiness monitoring service system includes a teaching terminal, an Internet of Things data acquisition device, and a sleepiness monitoring server. The teaching terminal is connected to the sleepiness monitoring server through an interface in the form of a Restful Service. The teaching terminal includes a teacher terminal and a student terminal, which are used to conduct teaching activities and start the sleepiness monitoring service and receive sleepiness monitoring results through the interface; the Internet of Things data acquisition device is connected to the sleepiness monitoring server through a wired or wireless network, including a visual acquisition device, an EEG acquisition device, and a heart rate acquisition device, which are used to collect posture data, brain wave data, and ECG signal data generated by students during online learning; the sleepiness monitoring server serves as a computing center, and includes startup and shutdown services for data acquisition equipment, a data processing module, and a fusion computing module.

[0057] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0058] 1) Compared with the existing online classroom scenario where teachers need to carefully watch the small window to detect whether students are dozing off, the present invention detects the drowsiness of a student through technical means and gives the teacher targeted feedback, so that the teacher can more conveniently grasp the student's class status.

[0059] 2) Compared with some current situations where learning, analysis, calculation, and teaching are performed through the same system, the drowsiness detection system proposed in this invention uses a separate fusion computing server to complete the analysis of students' drowsiness status and promptly feeds back the results to teachers through an interface, thereby improving computing efficiency and the timeliness of student status feedback.

[0060] 3) Compared with the existing technology that only uses single facial feature data to judge the student's fatigue status, the present invention increases the diversity of data (EEG + heart rate), uses multimodal data for fusion decision-making, and complements each other in the multimodal data, making the judgment result of the student's sleepiness state more accurate.

[0061] 4) Compared with the traditional DS evidence theory, the improved evidence theory method mentioned in the present invention improves the problem of multi-source data conflict and solves the paradox that violates common sense when the traditional DS evidence theory fuses high-conflict data. The advantage of the method proposed in the present invention is that even if high data conflict occurs during the fusion process, the fusion result will not violate common sense. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0063] Figure 2 Flowchart of the technical solution for the sleepiness monitoring method for multimodal students' online learning.

[0064] Figure 3 Schematic diagram of facial feature points around the eyelids.

[0065] Figure 4 Schematic diagram of feature points around the mouth. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0067] like Figure 1 As shown, the sleepiness monitoring system of the present invention as a whole includes a teaching terminal, an Internet of Things data acquisition device, and a sleepiness monitoring server. The teaching terminal is connected to the sleepiness monitoring server through an interface in the form of a Restful Service. The teaching terminal includes a teacher terminal and a student terminal, which can be any device with PC functions (such as a laptop, a tablet, etc.), used for conducting teaching activities and starting the sleepiness monitoring service and receiving sleepiness monitoring results through the interface. The Internet of Things data acquisition device is connected to the sleepiness monitoring server through a wired or wireless network, including a visual acquisition device, an EEG acquisition device, and a heart rate acquisition device, which is used to collect posture data, brain wave data, and electrocardiogram signal data generated by students when learning online. The sleepiness monitoring server serves as a computing center, including startup and shutdown services for data acquisition equipment, a data processing module, and a fusion computing module.

[0068] The operation process of the entire system is: the teacher opens the online classroom on the teacher side of the teaching terminal, the student enters the online classroom on the student side of the teaching terminal and wears the corresponding collection equipment, the teaching terminal initiates the start of the sleepiness monitoring service to the sleepiness monitoring server through the interface, and the sleepiness monitoring server starts the networked data collection equipment to collect student data after receiving the start signal. The various collected data are transmitted to the data processing module in the sleepiness monitoring server for calculation and processing before data fusion. After the processing is completed, the fusion calculation module performs data fusion and outputs the student sleepiness status after fusion calculation to the teaching terminal through the interface. At this time, the teacher can use the monitoring results to remind the students in the online classroom to pay attention and listen to the lecture, and the student end can also receive the corresponding sound reminder.

[0069] like Figure 2 As shown in the figure, the multimodal student online learning sleepiness monitoring method based on the improved DS evidence theory is implemented as follows:

[0070] S1. After the teacher starts the class, the teaching terminal sends a signal to the sleepiness monitoring service system. The system turns on the visual acquisition device, EEG device, and heart rate device for each student and starts collecting the student's online learning data.

[0071] S2: Based on preset time intervals, the student images captured by the visual acquisition device are input into a trained facial feature point extraction model. The obtained feature point coordinates around the eyes, nose, and mouth are used to calculate eye closure frequency data, sleepy nodding frequency data, and yawning frequency data. The student's EEG data collected by the EEG device is calculated to obtain the relative energy ratio of the brain waves within the time interval. The student's ECG signal data collected by the heart rate device is calculated to obtain the student's heart rate data within the time interval.

[0072] S3, based on the five types of data obtained, diagnose the local sleepiness state of the single-modal data respectively. The sleepiness state is divided into three categories: sleepy, awake, and unclear. The basic probability function of the three categories of states is assigned to the single-modal data, and five pieces of evidence are obtained.

[0073] S4, using the improved DS evidence theory, the evidence provided by the five types of data is integrated into decision-making.

[0074] S5, determine whether the fused student sleepiness value is greater than 0.5. If so, the student is deemed to be in a sleepy state and S6 is executed. Otherwise, the student is deemed to be awake and S1 is executed.

[0075] S6, the teacher is fed back the sleepiness of a student through the teaching terminal, and the student is reminded to pay attention and listen to the lecture. The system records the monitoring data.

[0076] S7, continuous monitoring, to determine whether the course has ended. If not, continue to execute S1. If the course has ended, shut down all monitoring devices and systems and perform data statistics.

[0077] There is also preparation work before S1, which is to build a facial feature point extraction model. The steps are:

[0078] S01, build a face sample library for students when studying, including face images of people with head-mounted EEG devices and public face libraries.

[0079] S02: Perform preprocessing such as data enhancement and labeling on the collected sample images.

[0080] S03, using the YOLO-V5 neural network to train and test in our sample library, through parameter adjustment and different image processing methods, continuous optimization, to obtain a high-precision facial feature extraction model.

[0081] The data collected and calculated by each device in S2 includes:

[0082] The S21 visual acquisition device collects students' online learning video data over a period of time, obtains images of each frame through background processing, and then inputs the images into a pre-trained facial feature extraction algorithm model to locate the faces in the images and extract the coordinates of the facial feature points. This can obtain three types of student learning posture data, namely:

[0083] (1) Eye-closing data: The average value of the aspect ratio (EAR) of the left and right eyes (EAR_avg) is calculated by the coordinates of the eye feature points (such as the facial feature points around the eyelids in the figure), as shown in formula (1). When EAR_avg is less than the threshold of 0.2, it is judged as closed eyes.

[0084]

[0085] Where p1-p6 represent the six coordinates of the eye feature points. i -p j || means p i to p j EAR_l represents the aspect ratio of the left eye, and EAR_r represents the aspect ratio of the right eye.

[0086] The PERCLOS algorithm is used to calculate the degree of eye closure of a person over a period of time. When a person is in a fatigued state, the number of closed frames increases significantly, and the PERCLOS parameter value increases accordingly. The calculation of the PERCLOS value is shown in formula (2). When its value is within the predetermined threshold range, according to P 80 The standard is that when the PERCLOS value is greater than or equal to 80%, it is determined to be a drowsy state.

[0087]

[0088] Among them, n is the total number of video frames within the detection time, i is the current frame number; Blink[i] is the number of closed-eye frames, is the total number of frames with closed eyes within the detection time.

[0089] (2) Nodding data: The feature points around the nose are calculated to determine whether the vertical change of the nose in two consecutive frames is greater than a threshold value. According to experience, the value is 0.03. If the nodding lasts for 3 seconds, it is judged as sleepy nodding and the person is in a drowsy state.

[0090] (3) Yawning data: Mouth Aspect Ratio (MAR) is based on the ratio of the distance between the feature points in the horizontal and vertical directions of the human mouth. Formula (3) can be used to determine whether the test person is opening his mouth and yawning. According to experience, when MAR is greater than or equal to 0.75, it is determined to be yawning and sleepy.

[0091]

[0092] The S22 EEG device collected brainwave data from students over a period of time. As sleepiness deepened, the complexity of the cerebral cortex decreased, the α-band frequency increased, and the β-band energy decreased. When the human body began to transition from fatigue to sleep, the θ-band frequency increased significantly. The range of different waveforms: θ waves (3-8Hz), α (8-12Hz), and β (12-27Hz). Therefore, the relative energy ratio of brainwaves can be used to represent sleepiness, as shown in formula (4). The smaller the r value, the greater the sleepiness. According to experience, when the r value is less than 1.46, it is determined to be a sleepy state.

[0093]

[0094] Where r is the spectral power distribution, and α, θ, and β represent the frequencies of α, θ, and β waves, respectively.

[0095] The S23 heart rate device collects ECG data from students over a period of time. As physical and mental fatigue deepens, the heart rate gradually slows, and so does the heart rate. Traditional methods for characterizing ECG signals are based on calculating the mean value during the RR peak in time-domain analysis, which lacks real-time performance. Frequency-domain calculations based on the Fast Fourier Transform (FFT) significantly improve analysis speed. The frequency of the ECG signal within a minute, or the heart rate, can be expressed using Equation (5). Based on its value within a predetermined threshold range, a drowsy state is determined when the f value is between 55 and 65, according to experience.

[0096]

[0097] Where X is the harmonic amplitude, F is the number of FFT sampling points, and fs is the FFT sampling frequency.

[0098] The specific introduction of the local evidence and basic probability distribution function provided by a single data in S3 is as follows:

[0099] S31 divides the sleepy state into three categories: A sleepy, B awake, and C unknown. It distributes the basic probability function of the three states to a single data (i.e., by calculating the probability value of the student's sleepy state in each data piece based on the proportion of sleepy states detected within a period of time), and defines the unclear judgment situations such as errors or missing data as C unknown state. During the period of data collection, a total of 5 pieces of evidence will be obtained. For example: the evidence provided by the closed-eye data is expressed as Z1: m1(A)=0.8, m1(B)=0.1, m1(C)=0.1, which means that the probability of the closed-eye data in this period of time judging the student as sleepy is 0.8, the probability of being awake is 0.1, and the probability of being unknown is 0.1. If there is a situation that may be sleepy but is not certain, use m i (AC) is represented, which will be mentioned in the examples. The evidence provided by the nodding data is represented by Z2: m2; the evidence provided by the yawning data is represented by Z3: m3; the evidence provided by the EEG data is represented by Z4: m4; and the evidence provided by the heart rate data is represented by Z5: m5. Using the improved DS evidence theory, the evidence provided by the five data types is fused to obtain the fused sleepiness level result m 1,2,3,4,5 :m(A)=a, m(B)=b, m(C)=c.

[0100] Regarding the improved DS evidence theory described in step S4 above, considering the paradox problem caused by the conflict between evidences, the improvement steps are as follows:

[0101] S41, using the Jousellme distance, calculate the distance between each two pieces of evidence. The distance calculation formula is as follows (6). Based on the distance, the similarity of each two pieces of evidence is calculated. The similarity calculation formula is as follows (7). Then, the adjacency matrix of the evidence relationship is constructed as follows:

[0102] Table 1.

[0103]

[0104] Where D is an N×N matrix, and N is the number of elements in the union of evidence i and evidence j. The elements in D are defined as: |·| represents the cardinality of set ·, that is, the number of elements in set ·, the ratio of the intersection and union of A and B.

[0105] Sim(m i ,m j )=1-d ij(m i ,m j ) (7)

[0106] The specific calculation of D is as follows:

[0107] Evidence source 1 is: m1(A)=0.41, m1(B)=0.29, m1(C)=0.30

[0108] Evidence source 3 is: m3(A)=0.58, m3(B)=0.07, m3(AC)=0.35;

[0109] Take the union of the set elements of the two evidence sources and get the new set {{A},{B},{C},{A,C}

[0110] vector and Represent the elements in the new collection in the order they appear:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] Table 1. Evidence relationship adjacency matrix

[0117]

[0118]

[0119] S42, calculation of evidence modification weight based on direct and indirect relationships. Here, the relationship strength of evidence (i.e., direct relationship) is defined as the sum of the similarities between the evidence body and other evidence bodies. The relationship strength calculation formula is as shown in formula (8).

[0120]

[0121] In the formula, i and j represent the number of evidence, N i is the set of evidence numbers other than evidence i.

[0122] The modification weight of the direct relationship between evidences is calculated using the relationship strength, as shown in formula (9).

[0123]

[0124] In the formula, n represents the total number of evidences, and here n is 5. Represents the sum of the relationship strengths of the five pieces of evidence.

[0125] S43, we construct a new set of evidence by removing specific evidence. The difference between the relationship strength in the original set (with specific evidence) and the relationship strength in the new set (without specific evidence) reflects the indirect relationship. The relationship coefficient of the indirect relationship of evidence is calculated as shown in Equation (10).

[0126]

[0127] Where VS j is to remove m i Front m j Relationship strength, VS j ′ is to remove m i After m j The modified weight of the indirect relationship is calculated by the indirect relationship coefficient λ of the evidence, as shown in formula (11).

[0128]

[0129] S44, integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight, as shown in formula (12),

[0130]

[0131] S45, finally the original evidence is weighted and modified by weighted averaging of evidence, as shown in formula (13).

[0132]

[0133] S46, after modifying the evidence body, the evidence is then combined using the DS combination rule to obtain the final fusion result.

[0134] The specific embodiments are as follows:

[0135] S1, the teacher clicks to start the class on the teaching terminal, the students enter the online classroom from the student terminal and wear the corresponding data collection equipment, the teaching terminal sends a signal to the sleepiness monitoring service system, the system turns on the visual collection equipment, EEG equipment, and heart rate equipment for each student, and starts collecting the students' online learning data.

[0136] S2, perform data processing, input the student images collected by the visual acquisition device into the trained facial feature point extraction model (YOLO-V5) according to the preset time interval, and calculate the eye closing frequency data, sleepy nodding frequency data, and yawning frequency data through the obtained feature point coordinates around the eyes, nose, and mouth. For facial image data, we can calculate the corresponding frequency data by dividing the eye closing frame, nodding frame, and yawning frame by the total number of frames in the preset time interval. Calculate the student EEG data collected by the EEG device to obtain the ratio data of the relative energy of the brain waves in the time interval. Calculate the student ECG signal data collected by the heart rate device to obtain the student's heart rate frequency data in the time interval.

[0137] S3, divide the sleepy state into three categories: A sleepy, B awake, C unknown, and assign the basic probability function of the three states to the single data.

[0138] a. Basic probability of closed-eye data: Take a 10-minute video of a student's face while learning online, captured by a camera, and 10,000 frames of eye image sample data. Divide this data into 100 batches of 100 frames each. Forty-one samples have a blink rate PERCLOS greater than or equal to 80%, 29 samples have a blink rate PERCLOS less than 80%, and some samples have no eye features detected. The basic probability distribution for these samples is 41% likely to be sleepy, 29% likely to be awake, and 30% unknown. The probability distribution function is: m1(A) = 0.41, m1(B) = 0.29, and m1(C) = 0.30.

[0139] b. The basic probability distribution function for yawning data is similar to that for eyes closed. Similarly, 10,000 frames of samples were taken. However, due to the camera's acquisition angle, no cases were detected where the student's mouth aspect ratio (MAR) exceeded 0.75. Furthermore, 10% of the data samples failed to detect feature points around the mouth. Therefore, the basic probability distribution function for yawning data is: m²(A) = 0.0, m²(B) = 0.90, and m²(C) = 0.10. This defective data demonstrates the present invention's ability to handle conflicting data.

[0140] For nodding detection, two frames of image data are required for detection, and two special data, EEG and heart rate, are used for the basic probability distribution of the drowsy state. Considering that the judgment of the unknown state by these three special data may miss the students who are actually drowsy, we define the proposition to be distributed as: A: drowsy, B: awake, AC: both unknown and drowsy.

[0141] c. Basic probability distribution of nodding data: Similarly, we take 10,000 frames of samples and divide them into 100 frames / portion, for a total of 100 portions. A sleepy nod is a 3-second period of lowering the head, which occurs in approximately 65 frames. However, since two frames are required for judgment, we calculate the cases where the nose changes along the vertical axis by more than 0.03 in two consecutive frames within the 100 frames. We find that 58 of the 100 samples detected a sleepy nod lasting 3 seconds, while only 7 samples detected a lowering of the head for no more than 3 seconds. The remaining samples are defined as both unclear and possibly sleepy. Therefore, the probability distribution for the proposition is: m3(A) = 0.58, m3(B) = 0.07, and m3(AC) = 0.35.

[0142] d. Basic probability distribution of EEG data: α, θ, and β waves within the detection period were divided into 100 samples of equal wavelength. By calculating the spectral power r, we determined whether the relative energy ratio of the three waves was within the sleepiness threshold. Of these 100 samples, 55 had r values ​​less than or equal to 1.46, 10 had r values ​​greater than 1.46, and 35 had missing data. Therefore, the probability distribution of the EEG data for the proposition was: m4(A) = 0.55, m4(B) = 0.10, and m4(AC) = 0.35.

[0143] e. Basic probability distribution of heart rate data: Take 10 minutes of sample data, with each minute as a sample, and test the heart rate every minute. It is found that the heart rate of students in 6 samples was between 55-65 beats / minute, and the heart rate in 1 minute was 78 beats / minute, which is between 65-120 beats / minute when awake. There were also 3 minutes with too low heart rates. It is judged that there may be equipment detection errors and other problems. However, this situation cannot be ignored. Therefore, the probability distribution of heart rate data for the proposition is: m5(A)=0.60, m5(B)=0.10, m5(AC)=0.30.

[0144] The evidence provided by the collected data on closing eyes, yawning, nodding, EEG, and heart rate is shown in the following table:

[0145]

[0146] S4, uses the improved DS evidence theory algorithm to perform fusion calculation on the five types of evidence obtained in S3.

[0147] S41, first calculate the Jousellmer distance between the evidences, and obtain the distance data between each evidence as follows:

[0148]

[0149] Then calculate the similarity between evidences and obtain the evidence relationship neighbor matrix. The data is shown in the following table

[0150]

[0151]

[0152] S42, using the similarity relationship between evidence, we first calculate the relationship strength of the evidence body. For the closed-eye evidence M1, our relationship strength is

[0153]

[0154] Similarly, for yawning M2, VS2=1.0716; for nodding M3, VS3=2.7635; for EEG M4, VS4=2.8239; and for heart rate M5, VS5=2.8055.

[0155] Then the relationship strength is used to calculate the modification weight of the direct relationship between the evidences.

[0156]

[0157] Similarly, wd2 = 0.0899; wd3 = 0.2319; wd4 = 0.2369; wd5 = 0.2354

[0158] S43: We construct a new set of evidence by removing specific evidence. The difference between the relationship strength in the original set (with specific evidence) and the relationship strength in the new set (without specific evidence) reflects the indirect relationship. The relationship coefficient of the indirect relationship of the evidence is calculated.

[0159] Specifically, calculate the indirect relationship coefficient of evidence M1, remove M1 to obtain a new set of evidence, and calculate the new relationship strength VS j ′, then calculate the indirect relationship coefficient,

[0160]

[0161] Then the modification weight of the indirect relationship of evidence M1 is calculated by the indirect relationship coefficient λ1 of the evidence.

[0162]

[0163] Similarly, we have wid2=0.0929; wid3=0.2293; wid4=0.2343; wid2=0.2343.

[0164] S44, integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight, and calculate for evidence M1, Similarly, wo(m2)=0.0388; wo(m3)=0.2469; wo(m4)=0.2578; wo(m5)=0.2562.

[0165] S45, finally, the original evidence is modified by weighted average of evidence, and the result can be calculated as Similarly, we can calculate that m(B)=0.1617; m(C)=0.3175.

[0166] S46, after modifying the evidence body, the DS combination rule is then used to combine the five types of sleepiness evidence four times to obtain the final fusion result, as shown in the following table.

[0167] <![CDATA[m1,m2]]> <h2 style=";text-align:left;direction:ltr"><![CDATA[m <h2 style=";text-align:left;direction:ltr"> 1, <h2 style=";text-align:left;direction:ltr"> m2,m3]]><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"><![CDATA[m <h2 style=";text-align:left;direction:ltr"> 1, <h2 style=";text-align:left;direction:ltr"> m2,m3,m4]]><h2 style=";text-align:left;direction:ltr"> <![CDATA[m1,m2,m3,m4,m5]]> m(A) = 0.8083 m(A) = 0.9252 m(A) = 0.9704 m(A) = 0.9881 m(B) = 0.0359 m(B) = 0.0087 m(B) = 0.0019 m(B) = 0.0004 m(C) = 0.1522 m(C) = 0.0661 m(C) = 0.0165 m(C) = 0.0115

[0168] However, traditional evidence theory does not process the case where the evidence m2(A)=0 provided by yawning, and the combined results are all m(A)=0, as shown in the table below. This obviously violates common sense. In contrast, the method of the present invention has a good effect on the fusion of such high-conflict data.

[0169] <![CDATA[m1,m2]]> <![CDATA[m1,m2,m3]]> <![CDATA[m1,m2,m3,m4]]> <h2 style=";text-align:left;direction:ltr"><![CDATA[m <h2 style=";text-align:left;direction:ltr"> 1, <h2 style=";text-align:left;direction:ltr"> m2,m3,m4,m5<h2 style=";text-align:left;direction:ltr"> m(A) = 0 m(A) = 0 m(A) = 0 m(A) = 0 m(B) = 0.8969 m(B) = 0. ​ ​ ​ ​ ​ ​

[0170] S5, judging whether the fused student sleepiness state value is greater than 0.5, according to the fusion result in S46, m(A)=0.9881>0.5, finally judging that the student's sleepiness state during this period is A: sleepy, and executing S6.

[0171] S6. After the drowsiness monitoring system calculates the fusion decision result, it transmits the student's drowsiness status to the teaching terminal through the interface and feeds back to the teacher on the teacher's screen. After checking which student is dozing off, the teacher can remind the student to pay attention and listen to the lecture. The system records the monitoring data throughout the process.

[0172] S7, continuous monitoring, to determine whether the course has ended. If not, continue to execute S1. If the course has ended, shut down all monitoring devices and systems and perform data statistics.

[0173] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A multimodal student sleepiness monitoring method based on evidence theory, characterized by: The steps include: S1: After the teacher starts the class, the teaching terminal sends a signal to the sleepiness monitoring service system. The system activates the visual acquisition device, EEG device, and heart rate device for each student to start collecting the student's online learning data. S2: Input the student images collected by the visual acquisition device into the trained facial feature point extraction model according to the preset time interval, and calculate the eye closing frequency data, sleepy nodding frequency data, and yawning frequency data based on the obtained feature point coordinates around the eyes, nose, and mouth; calculate the student's EEG data collected by the EEG device to obtain the ratio of the relative energy of the brain waves within the time interval; calculate the student's ECG signal data collected by the heart rate device to obtain the student's heart rate data within the time interval; S3, based on the five types of data obtained, diagnose the local sleepiness state of the single-modal data. The sleepiness state is divided into three categories: A sleepy, B awake, and C unknown. The basic probability function of the three categories of states is assigned to the single-modal data, and five pieces of evidence are obtained; The basic probability function distribution refers to calculating the probability value of the student's drowsy state in each data point by the proportion of drowsy states detected within a period of time, and defining the judgment of errors, missing data, or unclear states as unclear states C; the evidence provided by the eye closing frequency data is expressed as Z1: , , , which means that the probability of judging the student as sleepy based on the eye closing frequency data during this period is , the probability of being awake is , the probability of the unknown state is , the nodding frequency data provide evidence using Z2: The evidence provided by the yawning frequency data is expressed using Z3: The evidence provided by the data on the ratio of relative energy of brain waves is expressed as Z4: express; The evidence provided by the heart rate data is Z5: Using DS evidence theory, the evidence provided by the five types of data is fused to obtain the fused sleepiness level result. ; S4, using DS evidence theory, integrates the evidence provided by the five types of data into decision-making; The specific implementation of DS evidence theory in step S4 is as follows; S41, using Jousselme Distance, calculate the distance between each two pieces of evidence, and calculate the similarity between each two pieces of evidence based on the distance, and then construct the evidence relationship adjacency matrix; S42, based on the direct and indirect relationship between evidence, the modification weight is calculated. Here, the relationship strength of evidence is defined as the sum of the similarities between a certain evidence body and other evidence bodies. The calculation formula is as shown in formula (8); (8) Where, i and j Indicates the number of the evidence, Respectively refer to i Evidence and j evidence, To exclude i A collection of other evidence numbers other than evidence, is the similarity function; Using relationship strength to calculate the modification weight of direct relationship between evidences , as shown in formula (9); (9) In the formula, n represents the total number of evidences, and here n is 5. represents the sum of the relationship strengths of the five pieces of evidence; S43, by removing a specific body of evidence to form a new set of evidence, the difference between the relationship strength in the original evidence set and the relationship strength in the new evidence set reflects the indirect relationship, and the relationship coefficient of the indirect relationship of the evidence The calculation formula is as shown in formula (10); (10) In the formula is to remove forward The strength of the relationship, is to remove back The strength of the relationship, through the indirect relationship coefficient of evidence Calculating modification weights for indirect relationships , as shown in formula (11); (11) S44, integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight , as shown in formula (12), (12) S45, finally, the original evidence is weighted and modified by weighted average of evidence, as shown in formula (13); (13) S46, after modifying the evidence body, the evidence is then combined using the DS combination rule to obtain the final fusion result; S5, determining whether the fused student sleepiness value is greater than a threshold value, if so, the student is deemed to be in a sleepy state and executing S6, otherwise the student is deemed to be awake and executing S1; S6, the teacher is informed of the student's sleepiness through the teaching terminal, and the student is reminded to pay attention and listen to the lecture. The system records the monitoring data; S7, continuous monitoring, to determine whether the course has ended, if not, continue to execute S1, if the course has ended, shut down the system and perform data statistics.

2. The multimodal student drowsiness monitoring method based on evidence theory according to claim 1, characterized in that: The specific implementation method of obtaining the trained facial feature point extraction model in S2 is as follows; S21, build a face sample library for students when learning, including face images of people with head-mounted EEG devices and public face libraries; S22, preprocessing the collected sample images by data enhancement and labeling; S23 uses the YOLO-V5 neural network to train and test in the face sample library. Through parameter adjustment and different image processing methods, it continuously optimizes and obtains a high-precision facial feature extraction model.

3. The multimodal student sleepiness monitoring method based on evidence theory according to claim 1 is characterized in that: The specific implementation method of obtaining the eye closing frequency data, sleepy nodding frequency data, and yawning frequency data in S2 is as follows; (1) Eye closure frequency data: The aspect ratio of the left and right eyes is calculated by using the coordinates of the six feature points around the eyes. EAR Average value EAR_avg , as in formula (1), when EAR_avg When the value is less than the threshold value of 0.2, it is judged as closed eyes; (1) Where, - Represents the 6 coordinates of the eye feature points, express arrive The Euclidean distance, express arrive The Euclidean distance, express arrive The Euclidean distance, Indicates the aspect ratio of the left eye, is the aspect ratio of the right eye; use PERCLOS The algorithm calculates the degree of eye closure over a period of time. When a person is in a fatigued state, the number of eye closure frames increases significantly. PERCLOS As the parameter value increases, PERCLOS The calculation of the value is shown in formula (2), through its value in the predetermined threshold range, according to P 80 Standard, that is When the value is greater than or equal to 80%, it is determined to be a drowsy state; (2) in, n ' is the total number of video frames within the detection time, i' is the current frame number; is the number of frames with closed eyes, ; is the total number of frames with closed eyes within the detection time; (2) Frequency data of sleepy nodding: The feature points around the nose are calculated to determine whether the change amplitude of the nose along the vertical axis in two consecutive frames of images is greater than the threshold. When the nodding duration is 3 seconds, it is judged as sleepy nodding and the patient is in a sleepy state. (3) Yawning frequency data: mouth aspect ratio MAR It is based on the ratio of the distance between the feature points in the horizontal and vertical directions of the human mouth. Formula (3) is used to judge whether the test person is opening his mouth and yawning. According to experience, when MAR>=0.75, it is judged as yawning and sleepy. (3) Where, - They represent the coordinates of the six feature points around the mouth.

4. The multimodal student sleepiness monitoring method based on evidence theory according to claim 1, characterized in that: The specific implementation of step S4 is as follows: the brain wave relative energy ratio data is calculated using the following formula: (4) in, r is the spectral power distribution, α, θ, and β represent the frequencies of α wave, θ wave, and β wave, respectively. According to experience, when the r value is less than 1.46, it is determined to be a drowsy state.

5. The multimodal student drowsiness monitoring method based on evidence theory according to claim 1, characterized in that: The frequency of the ECG signal within 1 minute, i.e. the heart rate data, is calculated using the following formula: (5) Where X is the harmonic amplitude, F is the number of FFT sampling points, f s is the FFT sampling frequency, when f When the value is between 55-65, it is considered as drowsy state.

6. The multimodal student sleepiness monitoring method based on evidence theory according to claim 1, characterized in that: The distance calculation formula is as follows; (6) in, Indicates the i Evidence Hedi j Evidence The vector form of D is an N×N matrix, N is the evidence and evidence The number of elements in the union of the elements of D is defined as: , Representing a collection The cardinality of the set The number of elements in, the ratio of the intersection and union of A and B; The similarity calculation formula is as follows: (7)。 7. The multimodal student drowsiness monitoring method based on evidence theory according to claim 1, characterized in that: The sleepiness monitoring service system includes a teaching terminal, an Internet of Things data acquisition device, and a sleepiness monitoring server. The teaching terminal Restful Service The teaching terminal includes a teacher terminal and a student terminal, which are used for conducting teaching activities, starting the sleepiness monitoring service through the interface, and receiving the sleepiness monitoring results; the Internet of Things data acquisition device is connected to the sleepiness monitoring server through a wired or wireless network, including a visual acquisition device, an EEG acquisition device, and a heart rate acquisition device, which are used to collect the posture data, brain wave data, and ECG signal data generated by students during online learning; the sleepiness monitoring server serves as a computing center, including the startup and shutdown services of the data acquisition device, the data processing module, and the fusion computing module.

Images