Human intelligence-based state prediction method, edge computing device and medium

Abstract

The embodiment of the invention provides a state prediction method based on human intelligence, an edge computing device and a medium, and the method comprises the steps: obtaining multi-modal data which comprises at least two of the following data: video data, physiological data and voice data; performing feature extraction based on the video data to obtain target feature data corresponding to the video data; and / or performing feature extraction based on the physiological data and removing individual differences to obtain target feature data corresponding to the physiological data; and / or performing feature extraction based on the voice data and removing individual differences to obtain target feature data corresponding to the voice data; performing fusion processing based on the target feature data corresponding to each mode to obtain target fusion data; and predicting a personnel state based on the target fusion data.

Description

Technical Field

[0001] The present application relates to the field of state prediction technology, and specifically to a state prediction method, edge computing device and medium based on human intelligence. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, fatigue driving detection methods have become increasingly diverse. The research and application of fatigue driving detection technology has great practical significance for improving traffic safety. This technology can effectively reduce traffic accidents caused by fatigue driving and protect the lives of people and passengers.

[0003] In the prior art, the fatigue state of a person is mainly detected through video, and whether the person is fatigued is determined by identifying whether the person has closed his eyes for a long time based on the photos captured by the video.

[0004] However, using the existing technology methods, the reliability of fatigue driving detection is low. Summary of the invention

[0005] The embodiments of the present application provide a state prediction method, edge computing device and medium based on human intelligence.

[0006] In a first aspect, an embodiment of the present application provides a state prediction method based on human intelligence, comprising:

[0007] Acquiring multimodal data, the multimodal data comprising at least two of the following: video data, physiological data, and voice data;

[0008] Perform feature extraction based on video data to obtain target feature data corresponding to the video data; and / or, perform feature extraction based on physiological data to obtain first physiological feature data corresponding to the physiological data, perform individual difference removal processing on the first physiological feature data corresponding to the physiological data, and obtain target feature data corresponding to the physiological data; and / or, perform feature extraction based on voice data to obtain first voice feature data corresponding to the voice data, perform individual difference removal processing on the first voice feature data corresponding to the voice data, and obtain target feature data corresponding to the voice data;

[0009] Based on the target feature data corresponding to each modality, fusion processing is performed to obtain target fusion data;

[0010] The personnel status is predicted based on the target fusion data.

[0011] Optionally, performing individual difference removal processing on the first physiological characteristic data corresponding to the physiological data to obtain target characteristic data corresponding to the physiological data includes:

[0012] Extracting features from the resting physiological data to obtain second physiological feature data;

[0013] The quotient of the first physiological characteristic data and the second physiological characteristic data is obtained as the target characteristic data corresponding to the physiological data.

[0014] Optionally, the performing individual difference removal processing on the first voice feature data corresponding to the voice data to obtain target feature data corresponding to the voice data includes:

[0015] Extracting features from the speech data in the resting state to obtain second speech feature data;

[0016] The quotient of the cepstral coefficient of the first speech feature data and the cepstral coefficient of the second speech feature data is obtained as the target feature data corresponding to the speech data.

[0017] Optionally, the extracting features based on the video data to obtain target feature data corresponding to the video data includes:

[0018] Facial action features are extracted from the video data to obtain target feature data corresponding to the video data.

[0019] Optionally, the extracting features based on physiological data to obtain first physiological feature data corresponding to the physiological data includes:

[0020] The preprocessed physiological data is subjected to heart rate variability feature extraction and / or brain wave feature extraction to obtain first physiological feature data corresponding to the physiological data.

[0021] Optionally, the extracting features based on the voice data to obtain first voice feature data corresponding to the voice data includes:

[0022] The fundamental frequency feature, intensity feature and / or rhythm feature are extracted from the preprocessed speech data to obtain first speech feature data corresponding to the speech data.

[0023] Optionally, also include:

[0024] Normalizing the video data to obtain normalized video data;

[0025] Performing filtering processing on the physiological data to obtain filtered physiological data;

[0026] Perform anti-aliasing filtering on the voice data to obtain voice data after anti-aliasing filtering.

[0027] Optionally, the target feature data corresponding to each modality is fused to obtain the target fusion data, including:

[0028] The target feature data corresponding to each modality are spliced ​​to obtain the target fusion data.

[0029] In a second aspect, an embodiment of the present application provides a state prediction device based on human intelligence, comprising:

[0030] A processing module, for acquiring multimodal data, wherein the multimodal data includes at least two of the following: video data, physiological data and voice data; performing feature extraction based on the video data to obtain target feature data corresponding to the video data; and / or, performing feature extraction based on the physiological data to obtain first physiological feature data corresponding to the physiological data, performing individual difference removal processing on the first physiological feature data corresponding to the physiological data, and obtaining target feature data corresponding to the physiological data; and / or, performing feature extraction based on the voice data to obtain first voice feature data corresponding to the voice data, performing individual difference removal processing on the first voice feature data corresponding to the voice data, and obtaining target feature data corresponding to the voice data; performing fusion processing based on the target feature data corresponding to each modality to obtain target fusion data;

[0031] A prediction module is used to predict the status of a person based on the target fusion data.

[0032] Optionally, the processing module is specifically used to extract features from resting physiological data to obtain second physiological characteristic data; and obtain a quotient of the first physiological characteristic data and the second physiological characteristic data as target feature data corresponding to the physiological data.

[0033] Optionally, the processing module is specifically used to perform feature extraction on resting-state speech data to obtain second speech feature data; obtain the quotient of the cepstral coefficients of the first speech feature data and the cepstral coefficients of the second speech feature data as target feature data corresponding to the speech data.

[0034] Optionally, the processing module is also used to extract facial action features from the video data to obtain target feature data corresponding to the video data.

[0035] Optionally, the processing module is further used to extract heart rate variability characteristics and / or brain wave characteristics from the preprocessed physiological data to obtain first physiological characteristic data corresponding to the physiological data.

[0036] Optionally, the processing module is further used to extract fundamental frequency features, intensity features and / or rhythm features of the preprocessed speech data to obtain first speech feature data corresponding to the speech data.

[0037] Optionally, the processing module is specifically used to normalize the video data to obtain normalized video data; filter the physiological data to obtain filtered physiological data; and perform anti-aliasing filtering on the voice data to obtain anti-aliasing filtering voice data.

[0038] Optionally, the processing module is further used to perform splicing processing on the target feature data corresponding to each modality to obtain the target fusion data.

[0039] In a third aspect, an embodiment of the present application provides an edge computing device, comprising: a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the state prediction method based on human intelligence as described in any one of the first aspects are implemented.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the state prediction method based on human intelligence as described in any one of the first aspects are implemented.

[0041] The state prediction method based on human intelligence, edge computing device and medium provided in the embodiments of the present application obtain multimodal data, wherein the multimodal data includes at least two of the following: video data, physiological data and voice data; feature extraction is performed based on the video data to obtain target feature data corresponding to the video data; and / or feature extraction is performed based on the physiological data and individual differences are removed to obtain target feature data corresponding to the physiological data; and / or feature extraction is performed based on the voice data and individual differences are removed to obtain target feature data corresponding to the voice data; fusion processing is performed based on the target feature data corresponding to each modality to obtain target fusion data; and personnel status is predicted based on the target fusion data. The embodiments of the present application use multimodal data to predict personnel status, which is conducive to improving the reliability and accuracy of personnel status prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a state prediction method based on human intelligence provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0045] Figure 4A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0046] Figure 5 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0047] Figure 6 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0048] Figure 7 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0049] Figure 8 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0050] Fig. 9 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0051] Fig.10 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0052] Fig.11 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0053] Fig.12 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application;

[0054] Fig.13 A model structure diagram of a multimodal deep learning model provided in an embodiment of the present application;

[0055] Fig.14 A state prediction device based on human intelligence is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of the present application.

[0057] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited.

[0058] In the related art, the fatigue state of a person is detected based on the image in the video. For example, by dividing the collected video into multiple frames, the fatigue-related features such as the opening and closing state of the eyes and the head posture in the image are used to determine whether the person is tired. This method may cause overexposure of the video image under strong light, the items worn by the person may block the facial features, and the opening and closing degree of the upper and lower eyelids is not stable due to individual differences and the influence of external light. Therefore, the reliability of detection using the existing method is low.

[0059] In order to improve the reliability of detecting fatigue driving status, the present application provides a state prediction method, edge computing device and medium based on human factors intelligence. By acquiring multimodal data, the physiological data and voice data in the multimodal data are processed to remove individual differences, and then the multimodal data is fused to obtain target fusion data. The personnel status is then predicted based on the target fusion data to reduce the problem of missing feature information caused by traditional video-based detection of fatigue driving status or fatigue driving detection based on single data, thereby improving the reliability of fatigue detection.

[0060] The following embodiments of the present application can be executed by a vehicle, and a camera can be installed in the vehicle to collect facial videos of people while driving. A microphone device is added to the camera or a microphone device is added near the person to collect the person's voice data, and partition materials are used to reduce the interference of external traffic noise. During driving, the person can wear a physiological collection sensor to collect the person's physiological data. During the physiological data collection process, the person's resting state data in the early stage of the experiment will be collected to eliminate individual differences between people.

[0061] The following describes several specific embodiments of the state prediction method based on human intelligence. Figure 1 A flow chart of a state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Figure 1 As shown:

[0062] S11: Acquire multimodal data, where the multimodal data includes at least two of the following: video data, physiological data, and voice data.

[0063] Specifically, one possible implementation method is to obtain video data and physiological data; another possible implementation method is to obtain video data and voice data; another possible implementation method is to obtain physiological data and voice data; and yet another possible implementation method is to obtain video data, physiological data, and voice data.

[0064] The embodiment of the present application takes obtaining video data, physiological data and voice data as an example.

[0065] Among them, the video data refers to the facial video data of a person, and the facial video data includes the facial features of the person, such as the open and closed state of the eyes and the open and closed state of the mouth. The person can be a driver, a safety officer or a monitoring personnel, etc., and this application does not impose any restrictions on this.

[0066] Physiological data is data that can reflect the health status and physiological state of an individual, including electroencephalogram (EEG) and heart rate variability (HRV), etc. In fatigue driving detection, physiological data refers to biological indicators that can reflect the degree of fatigue of a person. The physiological data can be collected by various sensors and used to analyze the physiological and behavioral indicators of a person to determine whether he is in a fatigue state.

[0067] Voice data refers to the sound signal generated by a person when speaking. When a person is tired, the fundamental frequency of the voice will decrease. The fundamental frequency is the most basic frequency component in the voice signal and is related to the vibration frequency of the vocal cords. The intensity of the voice will also change. For example, a tired person may speak softly and the voice intensity will decrease.

[0068] When a person is driving a vehicle, the camera installed in the vehicle captures the person's facial video to obtain video data, and the person's voice data is collected through the on-board microphone. If the person wears a physiological sensor device, the physiological data of the person during driving can be obtained through the physiological sensor device.

[0069] The present application performs fusion processing on multimodal data to obtain target fusion data, see S12 and S13.

[0070] S12: Perform feature extraction based on the multimodal data to obtain target feature data corresponding to each modality.

[0071] Specifically, it includes at least two of the following:

[0072] Perform feature extraction based on the video data to obtain target feature data corresponding to the video data;

[0073] Perform feature extraction based on the physiological data to obtain first physiological feature data corresponding to the physiological data, and perform individual difference removal processing on the first physiological feature data corresponding to the physiological data to obtain target feature data corresponding to the physiological data;

[0074] Feature extraction is performed based on the voice data to obtain first voice feature data corresponding to the voice data, and individual difference removal processing is performed on the first voice feature data corresponding to the voice data to obtain target feature data corresponding to the voice data.

[0075] The embodiment of the present application takes feature extraction based on video data, physiological data and voice data as an example.

[0076] S13: Perform fusion processing based on the target feature data corresponding to each modality to obtain target fusion data.

[0077] Specifically, the target feature data corresponding to each modality are spliced ​​to obtain the target fusion data.

[0078] The target feature data extracted from the video data is flattened to one-dimensional data through the Flatten operation. Correspondingly, the target feature data extracted from the physiological data is flattened to one-dimensional data through the Flatten operation. The target feature data extracted from the voice data is flattened to one-dimensional data through the Flatten operation, and three one-dimensional data are obtained.

[0079] For example, if the dimension of a feature map is [32, 128, 10, 10], assuming there are 32 samples, each sample has 128 channels, and the feature map size of each channel is 10x10, the feature matrix size after Flatten will be [32, 128*10*10], that is, [32, 12800].

[0080] The three one-dimensional features are concatenated to obtain a fusion feature. For example, if the feature vectors of each modality are: the feature vector of video data: [1, 2, 3, 4, 5], the feature vector of physiological data: [6, 7, 8], and the feature vector of speech data: [9, 10, 11, 12], then the concatenated feature vector is: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].

[0081] S14: Predict personnel status based on target fusion data.

[0082] Among them, the prediction result is two outputs, which means that the fatigue state of the person is divided into two categories, including: fatigue state and non-fatigue state. Whether the person is in a fatigue state is determined by the probability of the prediction result, that is, assuming that the probability of the fatigue state is greater than the probability of the non-fatigue state, it is determined that the person is in a fatigue state.

[0083] In this embodiment, multimodal data is obtained, and the multimodal data includes at least two of the following: video data, physiological data, and voice data; the multimodal data is fused to obtain target fusion data; wherein, video data can monitor facial expressions and head postures of personnel, physiological data such as heart rate and brain waves can reflect the physiological state of personnel, and voice data can analyze the voice changes of personnel. By combining multiple data sources, fatigue signs of personnel can be more comprehensively captured, so that the use of multimodal data to predict the status of personnel is conducive to improving the reliability and accuracy of personnel status prediction. .

[0084] Figure 2 A flow chart of another state prediction method based on human intelligence provided in an embodiment of the present application is as follows: Figure 3 As shown, Figure 2 is Figure 1 On the basis of the illustrated embodiment, further, a possible implementation method of removing individual differences from the first physiological characteristic data corresponding to the physiological data in S12 is as follows:

[0085] S21: Extract features from the resting physiological data to obtain second physiological feature data.

[0086] Among them, resting state data refers to the physiological signal data collected when the subjects remain in a relaxed state without any specific tasks or stimulation.

[0087] When the person is in a relaxed state, resting physiological data is collected and feature extraction is performed to obtain second physiological characteristic data. Before feature extraction, the resting physiological data is subjected to the same time alignment, data slicing, data preprocessing and other operations as before obtaining the first physiological characteristic data.

[0088] S22: performing individual difference removal processing based on the first physiological characteristic data and the second physiological characteristic data to obtain target characteristic data corresponding to the physiological data.

[0089] Individual differences refer to differences in personality, age, physiology, psychology, etc., which can cause different physiological signals of different people to show different characteristics under the same conditions. There are differences in physiological basis between different individuals.

[0090] Specifically, the quotient of the first physiological characteristic data and the second physiological characteristic data is obtained as the target characteristic data corresponding to the physiological data. The calculation formula is:

[0091]

[0092] Wherein, SF is the first physiological characteristic data corresponding to the physiological data, SFb is the second physiological characteristic data, S norm It is the target feature data corresponding to the physiological data.

[0093] In this embodiment, feature extraction is performed on resting physiological data to obtain second physiological feature data, and individual difference removal processing is performed based on the first physiological feature data and the second physiological feature data to obtain target feature data corresponding to the physiological data. By removing individual differences in physiological data, misjudgment caused by individual differences can be reduced, thereby improving the accuracy of the detection results.

[0094] Figure 3 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, Figure 3 is Figure 1 On the basis of the embodiment shown, further, the method of removing individual differences from the first voice feature data corresponding to the voice data in S12 may be implemented as follows:

[0095] S31: Extract features from the resting-state speech data to obtain second speech feature data.

[0096] When the person is in a relaxed state, the resting voice data is collected and feature extraction is performed to obtain the second voice feature data. Before feature extraction is performed on the resting voice data, the same time alignment, data slicing and data preprocessing as before obtaining the first voice feature data are performed.

[0097] S32: Perform individual difference removal processing based on the first voice feature data and the second voice feature data to obtain target feature data corresponding to the voice data.

[0098] Specifically, the quotient of the cepstral coefficient of the first speech feature data and the cepstral coefficient of the second speech feature data is obtained as the target feature data corresponding to the speech data.

[0099] When removing individual differences based on the first voice feature data, the first voice feature data and the second voice feature data can be subjected to linear prediction coding (LPC) to obtain LPC coefficients, which can describe the frequency characteristics of the voice feature data. After the LPC coefficients are converted into cepstral coefficients, the cepstral coefficients are subjected to difference removal to achieve vocal tract length normalization, thereby removing the different vocal tract lengths of different speakers, which leads to individual differences. The calculation formula is:

[0100]

[0101] Wherein, n is the order of the cepstral coefficient, C(n) is the cepstral coefficient of the first speech feature data, α is a proportional coefficient related to the vocal tract length, and the proportional coefficient is calculated based on the static information data of the personnel, and C new (n) is the normalized cepstral coefficient.

[0102] The normalized cepstral coefficients are converted back to LPC coefficients, and the LPC coefficients are used to reconstruct the speech signal to obtain target feature data corresponding to the speech data.

[0103] In this embodiment, feature extraction is performed on the resting-state speech data to obtain second speech feature data, and individual difference removal processing is performed based on the first speech feature data and the second speech feature data to obtain target feature data corresponding to the speech data. By removing the individual differences in the speech data, misjudgment caused by individual differences can be reduced, thereby improving the accuracy of the detection results.

[0104] Figure 4 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, Figure 4 is Figure 1 On the basis of the illustrated embodiment, further, in S12, feature extraction is performed based on the video data to obtain target feature data corresponding to the video data in a possible implementation manner as follows:

[0105] S41: Extract facial action features from the video data to obtain target feature data corresponding to the video data.

[0106] Optionally, based on the Slowfast neural network, target feature data corresponding to the video data is obtained, that is, facial action features are extracted from the video data based on the Slowfast neural network to obtain target feature data corresponding to the video data.

[0107] In this embodiment, facial action features are extracted from video data to obtain target feature data corresponding to the video data, including the opening and closing status of the eyes and the opening and closing status of the mouth, so as to determine whether a person has actions such as closing his eyes and yawning, thereby providing a basis for subsequent multimodal data fusion.

[0108] Figure 5 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, Figure 5 is Figure 1 On the basis of the illustrated embodiment, further, in S12, feature extraction is performed based on physiological data to obtain a possible implementation method of first physiological feature data corresponding to the physiological data as follows:

[0109] S51: extracting heart rate variability characteristics and / or brain wave characteristics from the preprocessed physiological data to obtain first physiological characteristic data corresponding to the physiological data.

[0110] Optionally, the first physiological characteristic data is obtained based on a 3D convolutional neural network, that is, the 3D convolutional neural network can be used to extract heart rate variability characteristics and / or brain wave characteristics from the preprocessed physiological data to obtain the first physiological characteristic data corresponding to the physiological data.

[0111] Among them, when the person is relaxed, HRV increases, and when the person is in a fatigued state, HRV decreases. In the fatigued state, the power spectrum of the EEG signal will change. For example, the energy of slow waves (such as δ and θ waves) will increase when fatigued, while fast waves (such as α and β waves) will decrease accordingly.

[0112] In this embodiment, by extracting heart rate variability characteristics and / or brain wave characteristics from preprocessed physiological data and acquiring first physiological characteristic data corresponding to the physiological data, characteristic data of heart rate variability and / or brain waves of a person in different states can be obtained, and combined with state prediction based on the physiological data, a more comprehensive evaluation is provided, thereby improving the reliability of detection.

[0113] Figure 6 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, Figure 6 is Figure 1 On the basis of the illustrated embodiment, further, in S12, feature extraction is performed based on the voice data to obtain the first voice feature data corresponding to the voice data in a possible implementation manner as follows:

[0114] S61: extracting fundamental frequency features, intensity features and / or rhythm features from the preprocessed speech data to obtain first speech feature data corresponding to the speech data.

[0115] Optionally, the first speech feature data is obtained based on a probabilistic neural network, that is, the probabilistic neural network can be used to extract the fundamental frequency feature, intensity feature and / or rhythm feature of the preprocessed speech data to obtain the first speech feature data corresponding to the speech data.

[0116] When in a fatigue state, the fundamental frequency of speech will decrease. The fundamental frequency is the most basic frequency component in the speech signal, and the fundamental frequency is related to the vibration frequency of the vocal cords. In a fatigue state, the muscle control ability of the vocal cords decreases, the vibration frequency slows down, and the fundamental frequency decreases.

[0117] The intensity of speech may also change. For example, when fatigued, speech may become softer and less intense.

[0118] Prosody includes aspects such as intonation, stress, and rhythm. When you are tired, your intonation may vary less. When you speak normally, you use the ups and downs of your intonation to express emotion and emphasize key points, but when you are tired, your intonation may become more monotonous.

[0119] In this embodiment, by extracting the fundamental frequency features, intensity features and / or rhythm features of the preprocessed speech data, first speech feature data corresponding to the speech data is obtained, and speech data of a person in different states is obtained. The speech data of a person can be obtained without affecting the normal work of the person, thereby analyzing the fatigue state of the person and providing a basis for subsequently obtaining multimodal target fusion data.

[0120] Figure 7 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, Figure 7 is Figure 1 Based on the embodiment shown, before executing S12, the following may also be included:

[0121] S71: Preprocess the multimodal data to obtain preprocessed multimodal data.

[0122] Specifically, the video data is normalized to obtain normalized video data; the physiological data is filtered to obtain filtered physiological data; and the voice data is anti-aliased filtered to obtain anti-aliased voice data.

[0123] In this embodiment, the multimodal data is preprocessed to obtain preprocessed multimodal data, which improves data quality, enhances model performance and accuracy, and thus improves the overall effectiveness of the human intelligence state prediction method.

[0124] Figure 8 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, Figure 8 is Figure 7 On the basis of the illustrated embodiment, further, a possible implementation manner of S71 is as follows:

[0125] S81: performing normalization processing on the video data to obtain normalized video data.

[0126] The time-divided video data is normalized, the video data is divided into separate frames according to time, pixel values ​​are read for each frame of video image data, each pixel value is divided by 255, and the range of pixel values ​​is converted from [0, 255] to [0.0, 1.0] to obtain the first video data.

[0127] S82: Filter the physiological data to obtain filtered physiological data.

[0128] Different frequency bands are used for different physiological data, and bandpass filtering is performed on the physiological data to filter out noise signals. For example, the alpha wave in the EEG signal is related to the state of relaxation, and the theta wave is related to the state of sleepiness and fatigue. The EEG signal is filtered using a frequency band range of 1 to 50 to obtain an EEG signal with an effective frequency band, which means obtaining a specific frequency band related to fatigue, so that the fatigue state of a person can be detected more accurately.

[0129] S83: Perform anti-aliasing filtering on the voice data to obtain voice data after anti-aliasing filtering.

[0130] The speech signal is filtered using an anti-aliasing filter, which is a low-pass filter used to limit the bandwidth of the input signal, reduce the folding of high-frequency components into the low-frequency area, reduce signal aliasing, and thus improve the quality of the speech data.

[0131] In this embodiment, the video data is normalized to reduce the impact of uneven data distribution; the physiological data is filtered to reduce the noise in the physiological data and improve the clarity and accuracy of the signal; the voice data is anti-aliasing filtered to improve the quality of the voice signal, making the features in the voice more obvious, which is helpful for the identification of fatigue status.

[0132] Fig. 9 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, Fig. 9 is Figure 7 Based on the embodiment shown, before executing S71, the following steps may also be included:

[0133] S70: Perform time alignment and data slicing processing on the multimodal data.

[0134] Collect facial movement videos of people while driving. Wear physiological sensor devices to collect physiological data of people. Use microphones to record voice data of people while driving.

[0135] The collected data is time-aligned according to the timestamp of the collection. For example, assuming that the video data sampling rate is 30, the physiological data sampling rate is 256, and the speech sampling rate is 8000Hz, find 30 images collected by the video, 256 physiological signal sampling data, and 8000 sampling points of the speech signal in the same time to achieve time alignment.

[0136] The video data collected by the camera is divided into time segments. For example, the entire video stream data can be divided into 10-second segments, and each segment is marked as Vi, where i = 1, 2, 3, ..., n; the entire physiological data is divided into physiological data with the same time length as the video data, that is, one segment is divided every 10 seconds; the voice signal is a plurality of 8000*10 sample points.

[0137] In this embodiment, the multimodal data is time aligned and data sliced. The time alignment makes the data from different modalities match in time series, which improves the accuracy of data analysis. The data slice process can divide the continuous data stream into discrete segments, so that features can be extracted from each segment and analyzed, and the change of fatigue state over time can be observed more carefully, which enhances the correlation between the features of different modalities.

[0138] Fig.10 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, Fig.10 is Figure 1 On the basis of the illustrated embodiment, further, a possible implementation manner of S14 is as follows:

[0139] S141: Input the target fusion data into the target prediction model to obtain the personnel status.

[0140] The target fusion data is input into the target prediction model to predict the driver's fatigue state. The output of the target prediction model is a fatigue state or a non-fatigue state, thereby obtaining the personnel state.

[0141] In this embodiment, the target fusion data is input into the target prediction model to obtain the personnel status, and the feature data of multiple modalities are integrated, thereby improving the reliability of model prediction.

[0142] Fig.11 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, Fig.11 is Fig.10 Based on the embodiment shown, before S141, the following may also be included:

[0143] S140: The process of model training.

[0144] The multimodal deep learning model is trained based on the training sample set until the multimodal deep learning model converges to obtain the target prediction model.

[0145] The training data set includes: video data, physiological data and voice data. The multimodal model is trained using the training data set. The target fusion data of video data, physiological data and voice data are input into the fully connected layer to classify the driver's fatigue state. The fully connected layer maps these features to the output space of the classification task. For example, if there are 2 categories, the output dimension of the fully connected layer is 2, which can be fatigue state and non-fatigue state. The model parameters are adjusted by optimization algorithms such as back propagation and gradient descent until the model converges to obtain the target prediction model.

[0146] In this embodiment, a multimodal deep learning model is trained based on a training sample set until the multimodal deep learning model converges to obtain the target multimodal deep learning model. Data can be obtained from three aspects: video, physiology, and speech, thereby enhancing the generalization ability of the target multimodal deep learning model under different conditions, thereby improving the reliability of model prediction.

[0147] Fig.12 A flowchart of another state prediction method based on human intelligence provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, Fig.12 is Fig.11 On the basis of the illustrated embodiment, further, a possible implementation of S140 is as follows:

[0148] S1201: Perform time alignment, data slicing and data preprocessing based on the collected video data, physiological data and voice data to obtain target feature data, first physiological feature data and first voice feature data corresponding to the video data.

[0149] The implementation method of this step is similar to that of S71 and will not be repeated here.

[0150] S1202: Remove individual differences based on the first physiological characteristic data to obtain target characteristic data corresponding to the physiological data.

[0151] The implementation method of this step is similar to that of S22 and will not be repeated here.

[0152] S1203: Remove individual differences based on the first voice feature data to obtain target feature data corresponding to the voice data.

[0153] The implementation method of this step is similar to that of S32 and will not be repeated here.

[0154] S1204: Generate a brain topography map based on the electroencephalogram signal in the target feature data corresponding to the physiological data.

[0155] Each frequency band of the EEG signal can generate a brain topography. For example, alpha waves can generate a brain topography, and theta waves can also generate a brain topography. The power spectra of alpha waves and theta waves are different in fatigue and non-fatigue states, and the distribution of brain topography also changes. For example, in a state of wakefulness and relaxation, alpha waves are mainly distributed in the occipital lobe area. However, as fatigue increases, alpha waves may spread to other brain areas, and their power may also increase in areas such as the frontal lobe.

[0156] When acquiring EEG signals, a certain number of electrodes are placed on the driver's scalp according to the physiological sensor device to obtain EEG signals. Optionally, one or more electrodes are selected as reference points, and the signals of other electrodes are measured relative to the reference points, such as earlobe reference, average reference, etc. Spatial interpolation is performed to estimate brain activity in areas that are not directly recorded, so that the acquired EEG signals are more comprehensive. The voltage value of each electrode or interpolation point is mapped to a color to form a heat map. For example, positive voltage is represented by warm colors and negative voltage is represented by cold colors, thereby obtaining a brain topography map of the EEG signal in the second physiological data.

[0157] S1205: Obtain a training sample set based on the target feature data corresponding to the video data, the brain topography, the target feature data corresponding to the physiological data, and the target feature data corresponding to the speech data.

[0158] S1206: Train a multimodal deep learning model based on the training sample set.

[0159] S1207: Determine whether the multimodal deep learning model converges. If so, execute S1208; if not, return to execute S1206, that is, return to execute training the multimodal deep learning model based on the training sample set.

[0160] A preset error domain value is set, and the error will become smaller and smaller during the training process of the multimodal deep learning model. When the error reaches the preset error threshold, it is determined whether the multimodal deep learning model has converged.

[0161] S1208: Determine the multimodal deep learning model as a target prediction model.

[0162] The converged multimodal deep learning model is determined as the target prediction model.

[0163] In this embodiment, by using multimodal target fusion data on the multimodal deep learning model, a converged multimodal deep learning model is determined as the target prediction model, thereby reducing dependence on single modality data and improving the reliability of prediction.

[0164] Fig.13 A model structure diagram of a multimodal deep learning model provided in an embodiment of the present application, such as Fig.13 As shown, the feature extraction part can be divided into four branches, namely: video feature extraction, brain topography feature extraction, heart rate variability feature extraction and speech feature extraction, among which EEG features and HRV features are both features of physiological data. After the features extracted by the four branches are reduced to one dimension, the fused features are obtained by feature concatenation. The fused features are used to classify the fatigue state of the personnel using a fully connected layer (FC). The output of FC can be set to two outputs, each of which represents the probability of the model predicting each category.

[0165] The processing method of feature extraction in this embodiment is similar to the feature extraction method in the above embodiments, and will not be repeated in this embodiment.

[0166] Fig.14 A state prediction device based on human factors intelligence provided for an embodiment of the present application includes: a processing module 1401 and a prediction module 1402, wherein the processing module 1401 is used to obtain multimodal data, and the multimodal data includes at least two of the following: video data, physiological data and voice data; feature extraction is performed based on the video data to obtain target feature data corresponding to the video data; and / or, feature extraction is performed based on the physiological data to obtain first physiological feature data corresponding to the physiological data, and individual difference removal processing is performed on the first physiological feature data corresponding to the physiological data to obtain target feature data corresponding to the physiological data; and / or, feature extraction is performed based on the voice data to obtain first voice feature data corresponding to the voice data, and individual difference removal processing is performed on the first voice feature data corresponding to the voice data to obtain target feature data corresponding to the voice data; fusion processing is performed based on the target feature data corresponding to each modality to obtain target fusion data; the prediction module 1402 is used to predict the state of a person based on the target fusion data.

[0167] Optionally, the processing module 1401 is specifically used to extract features from resting physiological data to obtain second physiological characteristic data; and obtain a quotient of the first physiological characteristic data and the second physiological characteristic data as target characteristic data corresponding to the physiological data.

[0168] Optionally, the processing module 1401 is specifically used to extract features from resting-state speech data to obtain second speech feature data; obtain the quotient of the cepstral coefficients of the first speech feature data and the cepstral coefficients of the second speech feature data as target feature data corresponding to the speech data.

[0169] Optionally, the processing module 1401 is further used to extract facial action features from the video data to obtain target feature data corresponding to the video data.

[0170] Optionally, the processing module 1401 is further used to extract heart rate variability characteristics and / or brain wave characteristics from the preprocessed physiological data to obtain first physiological characteristic data corresponding to the physiological data.

[0171] Optionally, the processing module 1401 is further used to extract fundamental frequency features, intensity features and / or prosody features from the preprocessed speech data to obtain first speech feature data corresponding to the speech data.

[0172] Optionally, the processing module 1401 is specifically used to normalize the video data to obtain normalized video data; filter the physiological data to obtain filtered physiological data; and perform anti-aliasing filtering on the voice data to obtain anti-aliasing filtering voice data.

[0173] Optionally, the processing module 1401 is further used to perform splicing processing on the target feature data corresponding to each modality to obtain the target fusion data.

[0174] The device of this embodiment can be used to execute the technical solutions of the above-mentioned method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0175] The embodiment of the present application also provides an edge computing device, the edge computing device comprising: a processor and a memory. The memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the following is achieved: Figures 1 to 13 The steps of the state prediction method based on human intelligence are shown.

[0176] The embodiment of the present application also provides a computer-readable storage medium, wherein a program or instruction is stored on the computer-readable storage medium, and when the program or instruction is executed by a processor, the following is implemented: Figures 1 to 13 The steps of the state prediction method based on human intelligence are shown.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0178] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A state prediction method based on human intelligence, characterized in that: include: Acquiring multimodal data, the multimodal data comprising at least two of the following: video data, physiological data, and voice data; Perform feature extraction based on the video data to obtain target feature data corresponding to the video data; And / or, performing feature extraction based on the physiological data to obtain first physiological feature data corresponding to the physiological data, and performing individual difference removal processing on the first physiological feature data corresponding to the physiological data to obtain target feature data corresponding to the physiological data; And / or, performing feature extraction based on the voice data to obtain first voice feature data corresponding to the voice data, and performing individual difference removal processing on the first voice feature data corresponding to the voice data to obtain target feature data corresponding to the voice data; Based on the target feature data corresponding to each modality, fusion processing is performed to obtain target fusion data; The personnel status is predicted based on the target fusion data.

2. The method according to claim 1, characterized in that The removing individual difference processing of the first physiological characteristic data corresponding to the physiological data to obtain the target characteristic data corresponding to the physiological data includes: Extracting features from the resting physiological data to obtain second physiological feature data; The quotient of the first physiological characteristic data and the second physiological characteristic data is obtained as the target characteristic data corresponding to the physiological data.

3. The method according to claim 1, characterized in that The performing individual difference removal processing on the first voice feature data corresponding to the voice data to obtain target feature data corresponding to the voice data includes: Extracting features from the speech data in the resting state to obtain second speech feature data; The quotient of the cepstral coefficient of the first speech feature data and the cepstral coefficient of the second speech feature data is obtained as the target feature data corresponding to the speech data.

4. The method according to claim 1, characterized in that: The extracting features based on the video data to obtain target feature data corresponding to the video data includes: Facial action features are extracted from the video data to obtain target feature data corresponding to the video data.

5. The method according to claim 1, characterized in that The extracting features based on the physiological data to obtain first physiological feature data corresponding to the physiological data includes: The preprocessed physiological data is subjected to heart rate variability feature extraction and / or brain wave feature extraction to obtain first physiological feature data corresponding to the physiological data.

6. The method according to claim 1, characterized in that The extracting features based on the voice data to obtain first voice feature data corresponding to the voice data includes: The fundamental frequency feature, intensity feature and / or rhythm feature are extracted from the preprocessed speech data to obtain first speech feature data corresponding to the speech data.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Normalizing the video data to obtain normalized video data; Performing filtering processing on the physiological data to obtain filtered physiological data; Perform anti-aliasing filtering on the voice data to obtain voice data after anti-aliasing filtering.

8. The method according to any one of claims 1 to 6, characterized in that: The target feature data corresponding to each modality is fused to obtain the target fusion data, including: The target feature data corresponding to each modality are spliced ​​to obtain the target fusion data.

9. An edge computing device, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the state prediction method based on human intelligence as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the state prediction method based on human intelligence as described in any one of claims 1 to 8 are implemented.

Images