A method, apparatus, device, and storage medium for audio and video fatigue prediction based on time-series modeling.
By using a time-series modeling-based audio-visual fatigue prediction method, facial and audio features are extracted and personalized training is performed, which solves the problem of ignoring individual differences in existing technologies and achieves more accurate fatigue level prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2023-08-18
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies ignore individual differences in fatigue prediction, leading to limitations in the analysis of dynamic changes in fatigue characteristics.
By using a time-series modeling approach, facial and audio temporal features are extracted from the audio and video to be tested. The fatigue level is obtained by using a pre-trained target feature extraction module and a fatigue prediction module, combined with personalized training.
It effectively reflects individual differences, improves the robustness and accuracy of fatigue prediction, and reduces the impact of individual differences.
Smart Images
Figure CN117058660B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of fatigue prediction technology, and in particular to an audio-visual fatigue prediction method, apparatus, device and storage medium based on time-series modeling. Background Technology
[0002] In related technologies, the driver's electroencephalogram (EEG) signals and facial expression image blocks are collected and embedded; and multi-head self-attention analysis is performed through a Transformer encoder (Transformer is an encoder-decoder architecture, consisting of an encoder and a decoder, both of which are superpositions of multi-head self-attention modules), ultimately achieving prediction and feedback on the driver's driving fatigue state.
[0003] Among related technologies, there is also a visual fatigue detection method based on a two-stream convolutional neural network. This method can construct a multimodal fatigue database by collecting eye-tracking data, facial data, depth data, and fatigue level labels, and then construct training sample sets, validation sample sets, and test sets based on this database. This type of method uses a one-dimensional convolutional neural network algorithm to construct a two-stream convolutional neural network model and uses gradient descent to train it, ultimately achieving online real-time detection of the operator's visual fatigue level.
[0004] However, humans exhibit certain characteristics over a period of time due to fatigue. The fatigue analysis methods based on global feature capture often ignore the dynamic changes of these characteristics, i.e., they ignore individual differences in fatigue characteristics, which has certain limitations. Summary of the Invention
[0005] To address or at least partially address the aforementioned technical problems, embodiments of this disclosure provide a method, apparatus, device, and storage medium for audio-visual fatigue prediction based on time-series modeling.
[0006] In a first aspect, embodiments of this disclosure provide a method for predicting audio-visual fatigue based on time-series modeling, the method comprising:
[0007] From the acquired audio and video to be tested, facial temporal maps and audio temporal features are extracted according to a preset frequency.
[0008] Determine the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module corresponding to the facial temporal map;
[0009] Input the facial temporal map into the pre-trained target facial feature extraction module, and output facial feature change information in the temporal domain;
[0010] The audio temporal features are input into a pre-trained target audio feature extraction module, which outputs audio feature change information in the temporal domain.
[0011] The facial feature change information in the time domain and the audio feature change information in the time domain are input into the pre-trained target fatigue prediction module, and the fatigue level corresponding to the audio and video to be tested is output.
[0012] In one possible implementation, the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module are trained through the following steps:
[0013] Acquire known audio and video data, and extract facial temporal maps and audio temporal features from the known audio and video data according to a preset frequency;
[0014] The facial temporal map and audio temporal features are input into the initial facial feature extraction module and the initial audio feature extraction module, respectively. The outputs of the facial feature extraction module and the audio feature extraction module are input into the initial target fatigue prediction module, which outputs the fatigue prediction result of the known audio and video.
[0015] By comparing the known fatigue level labels with the fatigue level prediction results of audio and video, the facial feature extraction module, audio feature extraction module and fatigue prediction module are trained based on the comparison results, resulting in the first facial feature extraction module, the first audio feature extraction module and the first fatigue prediction module.
[0016] The first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained individually to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module.
[0017] In one possible implementation, the personalized training of the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module includes:
[0018] Receive known audio and video from multiple target faces;
[0019] For each target face, extract facial temporal maps and audio temporal features from the known audio and video according to a preset frequency.
[0020] The facial temporal map and audio temporal features are input into the first facial feature extraction module and the first audio feature extraction module, respectively. The outputs of the first facial feature extraction module and the first audio feature extraction module are input into the first fatigue prediction module, and the fatigue level prediction result of the known audio and video is output.
[0021] By comparing the fatigue level labels of known audio and video with the fatigue level prediction results, the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained based on the comparison results, resulting in the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to each target face.
[0022] In one possible implementation, extracting facial timing images from the acquired audio-visual material to be tested according to a preset frequency includes:
[0023] Extract video clips containing facial information from the acquired audio and video to be tested;
[0024] For video clips containing facial information, convert them into facial timing maps of preset pixels according to a preset frequency.
[0025] In one possible implementation, extracting audio timing features from the acquired audio / video data at a preset frequency includes:
[0026] Using the OpenSMILE toolkit, eGeMAPS features were extracted from the acquired audio and video samples at a frequency of 2Hz, serving as audio temporal features.
[0027] In one possible implementation, the pre-trained target facial feature extraction module is a recurrent neural network including a two-layer GRU model. The step of inputting a facial temporal map into the pre-trained target facial feature extraction module and outputting temporal facial feature change information includes:
[0028] Using the Dlib library in Python, facial feature points are annotated on a time-series facial icon.
[0029] A temporal map containing facial feature points is used as input to a recurrent neural network to extract the temporal variation information of facial feature points, which is then used as temporal feature variation information.
[0030] In one possible implementation, the pre-trained target audio feature extraction module includes a two-layer GRU model. The step of inputting audio temporal features into the pre-trained target audio feature extraction module and outputting audio feature change information in the time domain includes:
[0031] The audio temporal features are input into a two-layer GRU model, which outputs information on the changes in audio features in the time domain.
[0032] Secondly, embodiments of this disclosure provide an audio / video fatigue prediction device based on time-series modeling, comprising:
[0033] The extraction module is used to extract facial temporal maps and audio temporal features from the acquired audio and video to be tested according to a preset frequency;
[0034] The determination module is used to determine the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to the facial temporal map;
[0035] The first input module is used to input the facial temporal map into the pre-trained target facial feature extraction module and output facial feature change information in the temporal domain.
[0036] The second input module is used to input audio temporal features into the pre-trained target audio feature extraction module and output audio feature change information in the temporal domain.
[0037] The third input module is used to input facial feature change information in the time domain and audio feature change information in the time domain into the pre-trained target fatigue prediction module, and output the fatigue level corresponding to the audio and video to be tested.
[0038] In one possible implementation, the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module in the determination module are trained through the following steps:
[0039] Acquire known audio and video data, and extract facial temporal maps and audio temporal features from the known audio and video data according to a preset frequency;
[0040] The facial temporal map and audio temporal features are input into the initial facial feature extraction module and the initial audio feature extraction module, respectively. The outputs of the facial feature extraction module and the audio feature extraction module are input into the initial target fatigue prediction module, which outputs the fatigue prediction result of the known audio and video.
[0041] By comparing the known fatigue level labels with the fatigue level prediction results of audio and video, the facial feature extraction module, audio feature extraction module and fatigue prediction module are trained based on the comparison results, resulting in the first facial feature extraction module, the first audio feature extraction module and the first fatigue prediction module.
[0042] The first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained individually to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module.
[0043] In one possible implementation, the determining module, which involves performing personalized training on the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module, includes:
[0044] Receive known audio and video from multiple target faces;
[0045] For each target face, extract facial temporal maps and audio temporal features from the known audio and video according to a preset frequency.
[0046] The facial temporal map and audio temporal features are input into the first facial feature extraction module and the first audio feature extraction module, respectively. The outputs of the first facial feature extraction module and the first audio feature extraction module are input into the first fatigue prediction module, and the fatigue level prediction result of the known audio and video is output.
[0047] By comparing the fatigue level labels of known audio and video with the fatigue level prediction results, the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained based on the comparison results, resulting in the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to each target face.
[0048] In one possible implementation, the extraction module, specifically the step of extracting facial time-series images from the acquired audio-visual material to be tested according to a preset frequency, includes:
[0049] Extract video clips containing facial information from the acquired audio and video to be tested;
[0050] For video clips containing facial information, convert them into facial timing maps of preset pixels according to a preset frequency.
[0051] In one possible implementation, the extraction module, wherein extracting audio temporal features from the acquired audio-visual material to be tested according to a preset frequency, includes:
[0052] Using the OpenSMILE toolkit, eGeMAPS features were extracted from the acquired audio and video samples at a frequency of 2Hz, serving as audio temporal features.
[0053] In one possible implementation, the pre-trained target facial feature extraction module is a recurrent neural network including a two-layer GRU model. In the first input module, inputting the facial temporal map into the pre-trained target facial feature extraction module and outputting temporal facial feature change information includes:
[0054] Using the Dlib library in Python, facial feature points are annotated on a time-series facial icon.
[0055] A temporal map containing facial feature points is used as input to a recurrent neural network to extract the temporal variation information of facial feature points, which is then used as temporal feature variation information.
[0056] In one possible implementation, the pre-trained target audio feature extraction module includes a two-layer GRU model. In the second input module, the step of inputting audio temporal features into the pre-trained target audio feature extraction module and outputting audio feature change information in the temporal domain includes:
[0057] The audio temporal features are input into a two-layer GRU model, which outputs information on the changes in audio features in the time domain.
[0058] Thirdly, embodiments of this disclosure provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
[0059] Memory, used to store computer programs;
[0060] When the processor executes the program stored in the memory, it implements the above-mentioned audio and video fatigue prediction method based on time-series modeling.
[0061] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the above-described audio-visual fatigue prediction method based on time-series modeling.
[0062] Compared with the prior art, the technical solutions provided in this disclosure have at least some or all of the following advantages:
[0063] The audio-visual fatigue prediction method based on temporal modeling described in this disclosure extracts facial temporal maps and audio temporal features from the acquired audio-visual data to be tested according to a preset frequency; determines a target facial feature extraction module, a target audio feature extraction module, and a target fatigue prediction module corresponding to the facial temporal maps; inputs the facial temporal maps into a pre-trained target facial feature extraction module, which outputs facial feature change information in the temporal domain; inputs the audio temporal features into a pre-trained target audio feature extraction module, which outputs audio feature change information in the temporal domain; inputs the facial feature change information and the audio feature change information in the temporal domain into a pre-trained target fatigue prediction module, which outputs the fatigue level corresponding to the audio-visual data to be tested. By utilizing the temporal change information of fatigue-related features in the video and audio, the method learns the fatigue feature changes in the audio-visual time domain, effectively reflecting the fatigue level of the target face, capturing individual difference information, shortening the inference time, and increasing robustness. Attached Figure Description
[0064] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0065] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0066] Figure 1 This illustration schematically shows a flowchart of an audio-visual fatigue prediction method based on time-series modeling according to an embodiment of the present disclosure;
[0067] Figure 2 A schematic diagram illustrates the structure of an audio-visual fatigue prediction device based on time-series modeling according to an embodiment of the present disclosure;
[0068] Figure 3 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation
[0069] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0070] See Figure 1 The embodiments of this disclosure provide an audio-visual fatigue prediction method based on time-series modeling, including the following steps:
[0071] S1. Extract facial temporal maps and audio temporal features from the acquired audio and video to be tested according to a preset frequency.
[0072] In this embodiment, the acquired audio and video to be tested is obtained through the following steps:
[0073] Using facial recognition technology, audio and video containing facial information are extracted from the acquired raw audio and video, which is then used as the audio and video to be tested. The Haar feature classifier in OpenCV (a cross-platform computer vision library) (which consists of Haar feature extraction, a discrete strong classifier, and a strong classification cascade) can be used as a method to extract audio and video containing facial information from the acquired raw audio and video.
[0074] In this embodiment, facial time-series images are extracted according to a preset frequency, including:
[0075] The acquired audio and video to be tested are processed into a time-sorted image sequence at a frame rate of 2Hz and the format is standardized, wherein the format can be 256*256 pixels.
[0076] S2, determine the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module corresponding to the facial time sequence map.
[0077] In this embodiment, determining the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module corresponding to the facial temporal map includes:
[0078] Compare the facial timing image with the pre-stored target face;
[0079] Determine the first target face corresponding to the facial timing map;
[0080] Based on the pre-stored correspondence between target faces and target modules, the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module corresponding to the first target face are determined as the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module corresponding to the facial time sequence map.
[0081] S3 inputs the facial temporal map into the pre-trained target facial feature extraction module and outputs facial feature change information in the temporal domain.
[0082] S4, input the audio temporal features into the pre-trained target audio feature extraction module, and output audio feature change information in the temporal domain;
[0083] S5 inputs the facial feature change information in the time domain and the audio feature change information in the time domain into the pre-trained target fatigue prediction module, and outputs the fatigue level corresponding to the audio and video to be tested.
[0084] In this embodiment, in step S2, the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module are trained through the following steps:
[0085] Acquire known audio and video data, and extract facial temporal maps and audio temporal features from the known audio and video data according to a preset frequency;
[0086] The facial temporal map and audio temporal features are input into the initial facial feature extraction module and the initial audio feature extraction module, respectively. The outputs of the facial feature extraction module and the audio feature extraction module are input into the initial target fatigue prediction module, which outputs the fatigue prediction result of the known audio and video.
[0087] By comparing the known fatigue level labels with the fatigue level prediction results of audio and video, the facial feature extraction module, audio feature extraction module and fatigue prediction module are trained based on the comparison results, resulting in the first facial feature extraction module, the first audio feature extraction module and the first fatigue prediction module.
[0088] The first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained individually to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module.
[0089] In this embodiment, the personalized training of the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module includes:
[0090] Receive known audio and video from multiple target faces;
[0091] For each target face, extract facial temporal maps and audio temporal features from the known audio and video according to a preset frequency.
[0092] The facial temporal map and audio temporal features are input into the first facial feature extraction module and the first audio feature extraction module, respectively. The outputs of the first facial feature extraction module and the first audio feature extraction module are input into the first fatigue prediction module, and the fatigue level prediction result of the known audio and video is output.
[0093] By comparing the fatigue level labels of known audio and video with the fatigue level prediction results, the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained based on the comparison results, resulting in the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to each target face.
[0094] In this embodiment, the Adam optimizer (Adaptive Moment Estimation, an optimization algorithm based on gradient descent) is used during the training module. Compared with stochastic gradient descent, Adam is more stable, has a certain degree of noise resistance when the batch size is small, and has low memory requirements.
[0095] In this embodiment, step S1, extracting facial timing images from the acquired audio-visual material to be tested according to a preset frequency, includes:
[0096] Extract video clips containing facial information from the acquired audio and video to be tested;
[0097] For video clips containing facial information, convert them into facial timing maps of preset pixels according to a preset frequency.
[0098] In this embodiment, step S1, extracting audio timing features from the acquired audio-visual material to be tested according to a preset frequency, includes:
[0099] Using the OpenSMILE toolkit, eGeMAPS features were extracted from the acquired audio and video samples at a frequency of 2Hz as audio temporal features. OpenSMILE is an open-source toolkit for audio feature extraction. The extracted eGeMAPS features (emotional and global acoustic features for multiple speaker recognition) are a feature set for speech emotion recognition, including 88 dimensions, which contain information on emotion, speech quality, fundamental frequency, and speech pitch.
[0100] In this embodiment, in step S3, the pre-trained target facial feature extraction module is a recurrent neural network including a two-layer GRU model. The step of inputting the facial temporal map into the pre-trained target facial feature extraction module and outputting temporal facial feature change information includes:
[0101] Using the Dlib library (a C++ open-source toolkit containing machine learning algorithms) from the Python library (a Python library is a collection of code in Python that performs a certain function and is provided to users), facial feature points are annotated on the facial temporal chart;
[0102] A temporal map containing facial feature points is used as input to a recurrent neural network to extract the temporal variation information of facial feature points, which is then used as temporal feature variation information.
[0103] In this embodiment, the Gated Recurrent Unit (GRU) neural network includes a two-layer GRU model and layer normalization. Since the facial time series image contains a two-dimensional temporal sequence of facial feature information, the GRU model is used to process the spatial and temporal features of facial features. A layer normalization layer is added, and the hidden layers of the GRU model enable the Dropout algorithm (the dropout algorithm randomly removes some neurons from the hidden layers during training, along with all corresponding inputs and outputs), which accelerates model convergence and prevents overfitting. Because the number of samples for a single target face is relatively small, the Dropout algorithm is added during neural network training, causing neurons in the network to stop working with a certain probability. This probability is set to 0.3 to avoid the network learning overly specific features, leading to overfitting. Due to the recurrent structure of the GRU model and the temporal nature of the data, layer normalization normalizes all features for each sample, making the network more stable and improving the model's generalization performance.
[0104] In this embodiment, the following parameter settings were used during the training of the recurrent neural network: a two-layer GRU network with 512 neurons in each layer, an initial learning rate of 0.09, and a learning rate decay of 10% every 50 rounds, for a total of 500 rounds of training.
[0105] In this embodiment, in step S4, the pre-trained target audio feature extraction module includes a two-layer GRU model. The step of inputting audio temporal features into the pre-trained target audio feature extraction module and outputting audio feature change information in the temporal domain includes:
[0106] The audio temporal features are input into a two-layer GRU model, which outputs information on the changes in audio features in the time domain.
[0107] In this embodiment, in step S5, the target fatigue prediction module includes a fully connected layer and an activation function layer. The step of inputting temporal facial feature change information and temporal audio feature change information into the pre-trained target fatigue prediction module and outputting the fatigue level corresponding to the audio / video to be tested includes:
[0108] The facial feature change information in the time domain and the audio feature change information in the time domain are concatenated and input into the fully connected layer and activation function layer of the target fatigue prediction module to obtain the final output result.
[0109] This disclosure proposes a fatigue level prediction method based on temporal modeling, which combines audio features and facial features. It utilizes the temporal variation information of individual fatigue features, effectively reducing the impact of individual differences on the prediction results.
[0110] See Figure 2 The embodiments of this disclosure provide an audio-visual fatigue prediction device based on time-series modeling, comprising:
[0111] Extraction module 11 is used to extract facial temporal maps and audio temporal features from the acquired audio and video to be tested according to a preset frequency;
[0112] The determination module 12 is used to determine the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to the facial temporal map;
[0113] The first input module 13 is used to input the facial temporal map into the pre-trained target facial feature extraction module and output facial feature change information in the temporal domain.
[0114] The second input module 14 is used to input audio temporal features into a pre-trained target audio feature extraction module and output audio feature change information in the temporal domain.
[0115] The third input module 15 is used to input facial feature change information in the time domain and audio feature change information in the time domain into the pre-trained target fatigue prediction module, and output the fatigue level corresponding to the audio and video to be tested.
[0116] In this embodiment, the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module in the determination module are trained through the following steps:
[0117] Acquire known audio and video data, and extract facial temporal maps and audio temporal features from the known audio and video data according to a preset frequency;
[0118] The facial temporal map and audio temporal features are input into the initial facial feature extraction module and the initial audio feature extraction module, respectively. The outputs of the facial feature extraction module and the audio feature extraction module are input into the initial target fatigue prediction module, which outputs the fatigue prediction result of the known audio and video.
[0119] By comparing the known fatigue level labels with the fatigue level prediction results of audio and video, the facial feature extraction module, audio feature extraction module and fatigue prediction module are trained based on the comparison results, resulting in the first facial feature extraction module, the first audio feature extraction module and the first fatigue prediction module.
[0120] The first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained individually to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module.
[0121] In this embodiment, the determination module, which involves performing personalized training on the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module, includes:
[0122] Receive known audio and video from multiple target faces;
[0123] For each target face, extract facial temporal maps and audio temporal features from the known audio and video according to a preset frequency.
[0124] The facial temporal map and audio temporal features are input into the first facial feature extraction module and the first audio feature extraction module, respectively. The outputs of the first facial feature extraction module and the first audio feature extraction module are input into the first fatigue prediction module, and the fatigue level prediction result of the known audio and video is output.
[0125] By comparing the fatigue level labels of known audio and video with the fatigue level prediction results, the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained based on the comparison results, resulting in the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to each target face.
[0126] In this embodiment, the extraction module, specifically the step of extracting facial time-series images from the acquired audio-visual material to be tested according to a preset frequency, includes:
[0127] Extract video clips containing facial information from the acquired audio and video to be tested;
[0128] For video clips containing facial information, convert them into facial timing maps of preset pixels according to a preset frequency.
[0129] In this embodiment, the extraction module, specifically the step of extracting audio temporal features from the acquired audio-visual material to be tested according to a preset frequency, includes:
[0130] Using the OpenSMILE toolkit, eGeMAPS features were extracted from the acquired audio and video samples at a frequency of 2Hz, serving as audio temporal features.
[0131] In this embodiment, the pre-trained target facial feature extraction module is a recurrent neural network including a two-layer GRU model. In the first input module, inputting the facial temporal map into the pre-trained target facial feature extraction module and outputting temporal facial feature change information includes:
[0132] Using the Dlib library in Python, facial feature points are annotated on a time-series facial icon.
[0133] A temporal map containing facial feature points is used as input to a recurrent neural network to extract the temporal variation information of facial feature points, which is then used as temporal feature variation information.
[0134] In this embodiment, the pre-trained target audio feature extraction module includes a two-layer GRU model. The second input module, which inputs audio temporal features into the pre-trained target audio feature extraction module and outputs audio feature change information in the temporal domain, includes:
[0135] The audio temporal features are input into a two-layer GRU model, which outputs information on the changes in audio features in the time domain.
[0136] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0137] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0138] In the above embodiments, any and more of the extraction module 11, determination module 12, first input module 13, second input module 14, and third input module 15 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. At least one of the extraction module 11, determination module 12, first input module 13, second input module 14, and third input module 15 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the extraction module 11, the determination module 12, the first input module 13, the second input module 14, and the third input module 15 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.
[0139] See Figure 3 The electronic device provided in the embodiments of this disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140.
[0140] Memory 1130 is used to store computer programs;
[0141] When processor 1110 executes the program stored in memory 1130, it implements the following audio-visual fatigue prediction method based on time-series modeling:
[0142] From the acquired audio and video to be tested, facial temporal maps and audio temporal features are extracted according to a preset frequency.
[0143] Determine the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module corresponding to the facial temporal map;
[0144] Input the facial temporal map into the pre-trained target facial feature extraction module, and output facial feature change information in the temporal domain;
[0145] The audio temporal features are input into a pre-trained target audio feature extraction module, which outputs audio feature change information in the temporal domain.
[0146] The facial feature change information in the time domain and the audio feature change information in the time domain are input into the pre-trained target fatigue prediction module, and the fatigue level corresponding to the audio and video to be tested is output.
[0147] The aforementioned communication bus 1140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.
[0148] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.
[0149] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1130 may also be at least one storage device located remotely from the aforementioned processor 1110.
[0150] The processor 1110 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0151] Embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the time-series modeling-based audio-visual fatigue prediction method described above.
[0152] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the audio / video fatigue prediction method based on time-series modeling according to embodiments of this disclosure.
[0153] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0154] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0155] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A method for predicting audio and video fatigue based on time-series modeling, characterized in that, The method includes: From the acquired audio and video to be tested, facial temporal maps and audio temporal features are extracted according to a preset frequency. Determine the target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module corresponding to the facial temporal map; The facial temporal map is input into a pre-trained target facial feature extraction module, which outputs facial feature change information in the temporal domain. This includes: using the facial temporal map containing facial feature points as input to a recurrent neural network, extracting the change information of facial feature points in the temporal domain, and using this as facial feature change information in the temporal domain. The audio temporal features are input into a pre-trained target audio feature extraction module, and the audio feature change information in the temporal domain is output. This includes: inputting the audio temporal features into a two-layer GRU model and outputting the audio feature change information in the temporal domain. The facial feature change information in the time domain and the audio feature change information in the time domain are input into the pre-trained target fatigue prediction module, and the output is the fatigue level corresponding to the audio and video to be tested. This includes: concatenating the facial feature change information in the time domain and the audio feature change information in the time domain, and inputting them into the fully connected layer and activation function layer of the target fatigue prediction module to obtain the final output result. The target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module are trained through the following steps: Acquire known audio and video data, and extract facial temporal maps and audio temporal features from the known audio and video data according to a preset frequency; The facial temporal map and audio temporal features are input into the initial facial feature extraction module and the initial audio feature extraction module, respectively. The outputs of the facial feature extraction module and the audio feature extraction module are input into the initial target fatigue prediction module, which outputs the fatigue prediction result of the known audio and video. By comparing the known fatigue level labels with the fatigue level prediction results of audio and video, the facial feature extraction module, audio feature extraction module and fatigue prediction module are trained based on the comparison results, resulting in the first facial feature extraction module, the first audio feature extraction module and the first fatigue prediction module. The first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained individually to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module. The personalized training of the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module includes: Receive known audio and video from multiple target faces; For each target face, extract facial temporal maps and audio temporal features from the known audio and video according to a preset frequency. The facial temporal map and audio temporal features are input into the first facial feature extraction module and the first audio feature extraction module, respectively. The outputs of the first facial feature extraction module and the first audio feature extraction module are input into the first fatigue prediction module, and the fatigue level prediction result of the known audio and video is output. By comparing the fatigue level labels of known audio and video with the fatigue level prediction results, the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained based on the comparison results, resulting in the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to each target face.
2. The method according to claim 1, characterized in that, The step of extracting facial time sequence images from the acquired audio and video to be tested according to a preset frequency includes: Extract video clips containing facial information from the acquired audio and video to be tested; For video clips containing facial information, convert them into facial timing maps of preset pixels according to a preset frequency.
3. The method according to claim 1, characterized in that, The step of extracting audio temporal features from the acquired audio and video to be tested according to a preset frequency includes: Using the OpenSMILE toolkit, eGeMAPS features were extracted from the acquired audio and video samples at a frequency of 2Hz, serving as audio temporal features.
4. The method according to claim 1, characterized in that, The method further includes: Using the Dlib library in Python, facial feature points are annotated on a time-series facial icon.
5. An audio-visual fatigue prediction device based on time-series modeling, characterized in that, include: The extraction module is used to extract facial temporal maps and audio temporal features from the acquired audio and video to be tested according to a preset frequency; The determination module is used to determine the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to the facial temporal map; The first input module is used to input the facial temporal map into the pre-trained target facial feature extraction module and output facial feature change information in the temporal domain. The second input module is used to input audio temporal features into the pre-trained target audio feature extraction module and output audio feature change information in the time domain. The third input module is used to input facial feature change information and audio feature change information in the time domain into the pre-trained target fatigue prediction module and output the fatigue level corresponding to the audio and video to be tested. The target facial feature extraction module, target audio feature extraction module, and target fatigue prediction module in the determination module are trained through the following steps: Acquire known audio and video data, and extract facial temporal maps and audio temporal features from the known audio and video data according to a preset frequency; The facial temporal map and audio temporal features are input into the initial facial feature extraction module and the initial audio feature extraction module, respectively. The outputs of the facial feature extraction module and the audio feature extraction module are input into the initial target fatigue prediction module, which outputs the fatigue prediction result of the known audio and video. By comparing the known fatigue level labels with the fatigue level prediction results of audio and video, the facial feature extraction module, audio feature extraction module and fatigue prediction module are trained based on the comparison results, resulting in the first facial feature extraction module, the first audio feature extraction module and the first fatigue prediction module. The first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained individually to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module. In the determining module, the personalized training of the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module to obtain the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module includes: Receive known audio and video from multiple target faces; For each target face, extract facial temporal maps and audio temporal features from the known audio and video according to a preset frequency. The facial temporal map and audio temporal features are input into the first facial feature extraction module and the first audio feature extraction module, respectively. The outputs of the first facial feature extraction module and the first audio feature extraction module are input into the first fatigue prediction module, and the fatigue level prediction result of the known audio and video is output. By comparing the known fatigue level labels and fatigue level prediction results of audio and video, the first facial feature extraction module, the first audio feature extraction module, and the first fatigue prediction module are trained based on the comparison results, and the target facial feature extraction module, the target audio feature extraction module, and the target fatigue prediction module corresponding to each target face are obtained. The first input module further includes: taking a facial temporal map containing facial feature points as input to a recurrent neural network, and extracting the change information of facial feature points in the temporal domain as facial feature change information in the temporal domain; The second input module further includes: inputting audio temporal features into a two-layer GRU model and outputting audio feature change information in the temporal domain; The third input module further includes: connecting the facial feature change information in the time domain and the audio feature change information in the time domain, and inputting them into the fully connected layer and activation function layer of the target fatigue prediction module to obtain the final output result.
6. The apparatus according to claim 5, characterized in that, In the extraction module, the step of extracting facial time sequence images from the acquired audio-visual material to be tested according to a preset frequency includes: Extract video clips containing facial information from the acquired audio and video to be tested; For video clips containing facial information, convert them into facial timing maps of preset pixels according to a preset frequency.
7. The apparatus according to claim 5, characterized in that, In the extraction module, the step of extracting audio temporal features from the acquired audio-visual material to be tested according to a preset frequency includes: Using the OpenSMILE toolkit, eGeMAPS features were extracted from the acquired audio and video samples at a frequency of 2Hz, serving as audio temporal features.
8. The apparatus according to claim 5, characterized in that, The device further includes: Using the Dlib library in Python, facial feature points are annotated on a time-series facial icon.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the audio-visual fatigue prediction method based on time-series modeling as described in any one of claims 1-4.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the audio-visual fatigue prediction method based on time-series modeling as described in any one of claims 1-4.