A method for detecting learning concentration, computer device and readable medium

Abstract

The present application discloses a method for detecting learning concentration, a computer device and a readable medium, the method comprising: obtaining a voiceprint signal and a head posture image of a subject to be detected; inputting the voiceprint signal and the head posture image into a trained multimodal fusion model to extract the current voiceprint emotional features and head posture features of the subject to be detected; the multimodal fusion model comprises a first network model and a second network model; the multimodal fusion model fuses and classifies the voiceprint emotional features output by the first network model and the head posture features output by the second network model to obtain the current voiceprint emotional state and head posture angle of the subject to be detected; generating a current concentration detection result of the subject to be detected based on the voiceprint emotional state and the head posture angle; the present invention realizes the complementarity of voiceprint signal and head posture image information, improves the accuracy of learning concentration detection in the context of multi-person audio aliasing, and provides objective support data for improving students' online learning concentration.

Description

Technical Field

[0001] The present application relates to the field of intelligent human-computer interaction technology, and more specifically, to a learning concentration detection method, computer device and readable medium that integrates voiceprint and head posture clues. Background Art

[0002] With the rapid development of educational informatization, artificial intelligence technology has been increasingly applied in teaching and learning. In recent years, online distance learning methods have become widely adopted. However, without close teacher supervision, how to implement remote or self-supervision to ensure learners' independent learning status and efficiency has become a pressing issue. Therefore, it is of great significance to adopt a real-time online learning concentration detection method to assist learners in online learning and thus improve their learning efficiency.

[0003] Learning concentration can be reflected in aspects such as head posture, voiceprint emotion, and body movements. When a learner is focused, their head should be facing the learning device, their voice should be neutral or positive, and their body movements should be minimal. However, during online learning, learners may listen quietly or look down at their books. Relying solely on this information makes it difficult to accurately determine a learner's current state of concentration, resulting in reduced accuracy in the test results. Summary of the Invention

[0004] In response to at least one defect or improvement need in the prior art, the present invention provides a learning concentration detection method, computer device and readable medium that integrate voiceprint and head posture clues, which can achieve information complementarity of different modalities. Its purpose is to improve the accuracy of voiceprint emotion and head posture recognition in complex backgrounds, thereby improving the accuracy of learning concentration judgment.

[0005] To achieve the above object, according to one aspect of the present invention, a method for detecting learning concentration is provided, the method comprising:

[0006] Acquire the voiceprint signal and head posture image of the object to be detected;

[0007] Input the voiceprint signal and head posture image into the trained multimodal fusion model to extract the current voiceprint emotion characteristics and head posture characteristics of the object to be detected;

[0008] The multimodal fusion model includes a first network model and a second network model; the first network model is trained using at least one audio sample with an emotion label, and the second network model is trained using at least one head posture image sample with a head posture label;

[0009] The multimodal fusion model fuses and classifies the voiceprint emotion features output by the first network model and the head posture features output by the second network model to obtain the current voiceprint emotion state and head posture angle of the object to be detected;

[0010] A current concentration detection result of the object to be detected is generated based on the voiceprint emotional state and the head posture angle.

[0011] Preferably, in the above-mentioned learning concentration detection method, generating the current concentration detection result of the subject to be detected based on the voiceprint emotional state and head posture angle includes:

[0012] Input the voiceprint emotional state and head posture angle into the learning concentration detection model, and calculate the current concentration state of the subject to be detected based on the head posture angle and the preset learning space;

[0013] The emotional state and concentration state of the voiceprint are combined to obtain the current concentration detection result of the object to be detected.

[0014] Preferably, in the above-mentioned learning concentration detection method, the training process of the first network model includes:

[0015] Obtaining a first audio sample with a first emotion label as an anchor sample, a second audio sample with the first emotion label as a positive sample, and a first audio sample with a second emotion label as a negative sample; wherein the second emotion label is different from the first emotion label, and the second audio sample and the first audio sample belong to different objects;

[0016] Inputting the anchor sample, positive sample and negative sample into the first network model, and generating emotion feature vectors corresponding to the anchor sample, positive sample and negative sample respectively through the first network model to be trained;

[0017] A first triple loss function is calculated based on the emotion feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, and the model parameters of the first network model to be trained are reversely adjusted until the first triple loss function is minimized to obtain a trained first network model.

[0018] Preferably, in the above-mentioned learning concentration detection method, the first triple loss function is calculated based on the emotion feature vectors corresponding to the anchor sample, the positive sample and the negative sample, specifically:

[0019] Calculating a first similarity between the emotion feature vectors of the anchor sample and the positive sample, and a second similarity between the emotion feature vectors of the anchor sample and the negative sample;

[0020] A first triple loss function is calculated according to the difference between the first similarity and the second similarity, where the first triple loss function is expressed as follows:

[0021]

[0022] in, represents the emotional feature vector of the anchor sample extracted by the first network model, represents the sentiment feature vector of the positive sample, The sentiment feature vector representing the negative sample;

[0023] represents the first similarity, represents the second similarity, ɑ is the first distance threshold set during the network optimization process; n c Indicates the number of training times.

[0024] Preferably, in the above-mentioned learning concentration detection method, the training process of the second network model includes:

[0025] Acquire a first head pose image sample with a first head pose label as an anchor sample, a second head pose image sample with the first head pose label as a positive sample, and a first head pose image sample with a second head pose label as a negative sample; wherein the second head pose label is different from the first head pose label, and the second head pose image sample and the first head pose image sample belong to different objects;

[0026] Inputting the anchor sample, the positive sample, and the negative sample into the second network model, and generating head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample respectively through the second network model to be trained;

[0027] A second triple loss function is calculated based on the head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, and the model parameters of the second network model to be trained are reversely adjusted until the second triple loss function is minimized to obtain a trained second network model.

[0028] Preferably, in the above-mentioned learning concentration detection method, the second triple loss function is calculated based on the head posture feature vectors corresponding to the anchor sample, the positive sample and the negative sample, specifically:

[0029] calculating a third similarity between the head pose feature vectors of the anchor sample and the positive sample, and a fourth similarity between the head pose feature vectors of the anchor sample and the negative sample;

[0030] A second triple loss function is calculated according to the difference between the third similarity and the fourth similarity, where the second triple loss function is expressed as follows:

[0031] L triplet (x n ,x a ,x p )=max(0,D(x a ,x n )-D(x a ,x p )+γ)

[0032]

[0033] Among them, (x n ,x a ,x p ) represents three head pose sample pairs, corresponding to negative samples, anchor samples and positive samples respectively;

[0034] f(x a ) represents the head posture feature vector of the anchor sample extracted by the second network model, f(x n ) represents the head pose feature vector of the negative sample, f(x p ) represents the head posture feature vector of the positive sample;

[0035] D(x a ,x p ) represents the third similarity, D(x a ,x n ) represents the fourth similarity;

[0036] γ represents the set second distance threshold.

[0037] Preferably, in the above-mentioned learning concentration detection method, the multimodal fusion model further includes an aggregation network and a classification network;

[0038] The aggregation network is used to aggregate the voiceprint emotion features output by the first network model and the head posture features output by the second network model to obtain an aggregated feature;

[0039] The classification network generates the current voiceprint emotional state and head posture angle of the object to be detected based on the aggregated features.

[0040] Preferably, in the above learning concentration detection method, before inputting the voiceprint signal and the head posture image into the trained multimodal fusion model, the method further includes:

[0041] Pre-processing the voiceprint signal using pre-emphasis, framing and windowing operations, and / or,

[0042] The bicubic interpolation algorithm is used to perform image scaling on the head posture image.

[0043] According to the second aspect of the present invention, a computer device is also provided, which includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs the steps of any one of the above-mentioned learning concentration detection methods.

[0044] According to the third aspect of the present invention, a computer-readable medium is also provided, which stores a computer program that can be executed by a computer device. When the computer program is run on the computer device, the computer device executes the steps of any of the above-mentioned learning concentration detection methods.

[0045] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0046] (1) The learning concentration detection method provided by the present invention uses voiceprint information and head posture images for learning concentration detection at the same time, adopts a novel multimodal fusion model to accurately extract voiceprint emotional features and head posture features, and fuses the voice emotional features and head posture features, thereby overcoming the limitation of the single detection dimension in traditional detection methods, reducing the problems of missed detection and false detection, and having important significance for improving learning quality and assisting teaching.

[0047] (2) The present invention adopts a triple loss function to train the multimodal fusion model. The triple loss function can ensure that the network model maximizes the intra-class similarity and minimizes the inter-class similarity during the training process, thereby making the trained network more robust.

[0048] (3) The present invention adds a temporal attention mechanism module to the multimodal network model to focus on more effective information, suppress the impact of background noise on model accuracy, and improve the robustness of the network model, thereby effectively improving the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A flowchart of a method for detecting learning concentration provided in this embodiment;

[0051] Figure 2 Schematic diagram of data acquisition in online learning environment;

[0052] Figure 3A schematic diagram of the network structure of the multimodal fusion model provided in this embodiment;

[0053] Figure 4 A schematic diagram of a learning concentration judgment mechanism provided in an embodiment of the present application;

[0054] Figure 5 It is a schematic diagram of the composition structure of the computer device provided in this embodiment. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0056] The terms "first," "second," "third," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0057] Figure 1 This is a flow chart of a method for detecting learning concentration provided by this embodiment, see Figure 1 As shown, the method mainly includes the following steps:

[0058] S1 obtains the voiceprint signal and head posture image of the object to be detected;

[0059] In this embodiment, the object to be detected is an object that is using a learning device to perform online learning. Figure 2 The data acquisition diagram in the online learning environment provided in this embodiment is as follows: Figure 2 As shown in the figure, in a specific example, a visible light camera is used to record the audio and video of the object to be detected. The visible light camera can automatically select the RGB image of the head posture of interest in the audio and video and extract the voiceprint signal that corresponds to the RGB image in time. The visible light camera can be integrated into the learning device, or a separate device can be used and set up on the learning device, or it can be placed in another area where it can collect valid voiceprint signals and head posture images of the object to be detected.

[0060] In an optional embodiment, the voiceprint signal and head posture image of the subject to be detected are preprocessed as needed. Specifically:

[0061] (1) The input voiceprint data is preprocessed by pre-emphasis, framing and windowing operations. In a specific example, the voiceprint data is framed with a frame length of 25 milliseconds and a frame interval of 10 milliseconds. The window function of the windowing operation is a Hamming window. For a discrete signal x[n] with a length of N, 0≤n≤N-1, the weight ω[n] corresponding to the Hamming window function can be expressed as:

[0062] (2) grayscale, geometric transformation, and image enhancement are performed on the head posture image, thereby enhancing relevant information in the head posture RGB image and eliminating useless information;

[0063] In a specific example, the bicubic interpolation algorithm is used to perform image scaling on head pose RGB image data. The bicubic interpolation algorithm involves 16 pixels for image scaling. (i′, j′) represents the pixel coordinates of the pixel to be calculated in the source image, including the decimal part. dx represents the decimal coordinate in the X direction, and dy represents the decimal coordinate in the Y direction. Bicubic interpolation is essentially the sum of the weighted convolutions of the 16 pixels in the image as the new pixel value, that is:

[0064]

[0065] Where m and n are the fractional parts of floating-point coordinates in the range [-1, 2), and R(x) represents the interpolation expression based on Bell distribution sampling:

[0066]

[0067] S2 inputs the voiceprint signal and head posture image into the trained multimodal fusion model to extract the current voiceprint emotion characteristics and head posture characteristics of the object to be detected;

[0068] Figure 3 is a schematic diagram of the network structure of the multimodal fusion model provided in this embodiment, such as Figure 3 As shown, the multimodal fusion model includes a first network model and a second network model; the first network model is trained using at least one audio sample with an emotion label, and the second network model is trained using at least one head posture image sample with a head posture label;

[0069] The multimodal fusion model fuses and classifies the voiceprint emotion features output by the first network model and the head posture features output by the second network model to obtain the current voiceprint emotion state and head posture angle of the object to be detected.

[0070] In this embodiment, the first network model is a residual convolution-excitation compression network (RCSENet), which is used to extract the emotional features of the voiceprint signal; it includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer;

[0071] In a specific example, the first feature extraction layer includes a one-dimensional convolution layer, a batch normalization layer, and a ReLu layer. First, a one-dimensional convolution layer with H convolution kernels, a size of L, and a step size of L / 2 is used to encode the original waveform x = R^(1×T) of the voiceprint signal to obtain the voiceprint feature, which is expressed as X = R^(H×K); where T represents the length of the input audio, and K = 2(TL) / L+1. The voiceprint feature is input into the batch normalization layer, and the batch normalization layer and the ReLu layer are used to ensure the stability of the voiceprint feature representation. The encoded voiceprint feature X is first input into a convolution layer with P convolution kernels and a size of 1×1 to adjust the appropriate channel size for the subsequent network structure.

[0072] The second feature extraction layer includes multiple stacked feature extractors to further extract features from the voiceprint feature representation output by the first feature extraction layer; Figure 3 In this embodiment, the second feature extraction layer has three cascaded feature extractors, which are denoted as CSE modules. Multiple cascaded CSE modules can increase the receptive field of the feature extractor, thereby capturing the long-term contextual information of the input audio. The voiceprint features output by the latter feature extractor complement the features output by the previous feature extractor in time sequence.

[0073] The third feature extraction layer is implemented based on the ResNet34 network. Its main function is to extract the embedding feature representation containing the voiceprint emotion of the object to be detected based on the voiceprint features output by the second feature extraction layer.

[0074] In a preferred embodiment, an attention mechanism module is further provided between the second feature extraction layer and the third feature extraction layer. The purpose of adding the attention mechanism module is to focus on more effective voiceprint information and suppress the impact of background noise on the model, thereby improving the robustness of the network model.

[0075] As a specific example, the attention mechanism module includes an average pooling layer, a maximum pooling layer, and a multi-layer perceptron; the voiceprint features output by the first feature extraction layer are input into the average pooling layer and the maximum pooling layer respectively to obtain the average pooling feature and the maximum pooling feature; the multi-layer perceptron aggregates the average pooling feature and the maximum pooling feature to obtain the temporal attention feature. The temporal attention mechanism can be expressed as the formula:

[0076] M C (X)=σ(MLP(AvgPool(X))+MLP(MaxPool(X)))

[0077] Among them, M C (X) represents the voiceprint feature after the temporal attention mechanism, MLP(*) represents the multi-layer perceptron, σ represents the Sigmoid activation function, AvgPool(*) represents the average pooling operation, and MaxPool(*) represents the maximum pooling operation, which can be simplified as follows:

[0078]

[0079] Where c represents the number of channels, ω1 and ω0 are the shared parameters of the average pooling layer and the maximum pooling layer, respectively.

[0080] The first network model is trained using at least one audio sample with an emotion label, and the training process includes:

[0081] Obtain a first audio sample with a first emotion label as an anchor sample, a second audio sample with the first emotion label as a positive sample, and the first audio sample with a second emotion label as a negative sample; wherein the second emotion label is different from the first emotion label, and the second audio sample and the first audio sample belong to different objects; in a specific example, the emotion labels include negative, positive, and neutral;

[0082] Inputting the anchor sample, positive sample and negative sample into the first network model, and generating emotion feature vectors corresponding to the anchor sample, positive sample and negative sample respectively through the first network model to be trained;

[0083] A first triple loss function is calculated based on the emotion feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, and the model parameters of the first network model to be trained are reversely adjusted until the first triple loss function is minimized to obtain a trained first network model.

[0084] In this embodiment, a triple loss function is used as the loss function of the first network model to perform model training; a first audio sample with an emotion label is randomly selected as an anchor sample. Selecting and anchoring samples The second audio sample with the same emotion label is used as a positive sample The second audio sample and the first audio sample should belong to different objects; select and anchor samples The first audio sample with different emotion labels is used as a negative sample Negative samples With anchor sample Belong to the same object, but with different emotion labels; the anchor sample, positive sample and negative sample are composed of three audio sample pairs The first network model is trained. For each sample, the first network model predicts and outputs the corresponding emotion feature vector. A triple loss function is calculated based on the emotion feature vectors corresponding to the anchor sample, positive sample, and negative sample. The model parameters of the first network model are optimized until the triple loss function is minimized, indicating that the first network model training is complete.

[0085] In a preferred embodiment, the triple loss function is calculated based on the emotion feature vectors corresponding to the anchor sample, the positive sample, and the negative sample as follows:

[0086] Calculate a first similarity between the emotion feature vectors of the anchor sample and the positive sample, and a second similarity between the emotion feature vectors of the anchor sample and the negative sample; and calculate a triple loss function based on the difference between the first similarity and the second similarity.

[0087] The triple loss function can ensure that the network model maximizes the intra-class similarity and minimizes the inter-class similarity during training, thereby making the trained network more robust. The process of optimizing the first network model can be expressed as:

[0088]

[0089] in, represents the emotional feature vector of the anchor sample extracted by the first network model, represents the sentiment feature vector of the positive sample, The sentiment feature vector representing the negative sample; represents the first similarity, represents the second similarity, and α is the first distance threshold set during the network optimization process; represents a three-input sample pair, and T represents the batch in the triplet loss.

[0090] The process of network optimization is to increase the similarity between anchor samples and positive samples, and reduce the similarity between anchor samples and negative samples. The calculation of the triple loss function can be expressed as:

[0091]

[0092] The distance between positive samples and anchor samples is continuously shortened relative to negative samples through the above triple loss function.

[0093] In this embodiment, the second network model is an efficient Fisher matrix distribution network (E-FMDNet), which is used to extract head posture image features of head posture RGB image data; it includes a fourth feature extraction layer, a fully connected layer and an unconstrained matrix layer; the preprocessed head posture RGB image data is subjected to feature extraction through the fourth feature extraction layer, and the extracted head posture features are input into a fully connected layer and then passed through the unconstrained matrix to obtain the Fisher matrix distribution of the head posture image.

[0094] The second network model is trained using at least one head posture image sample with a head posture label, and the training process includes:

[0095] Acquire a first head pose image sample with a first head pose label as an anchor sample, a second head pose image sample with the first head pose label as a positive sample, and a first head pose image sample with a second head pose label as a negative sample; wherein the second head pose label is different from the first head pose label, and the second head pose image sample and the first head pose image sample belong to different objects;

[0096] Inputting the anchor sample, the positive sample, and the negative sample into the second network model, and generating head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample respectively through the second network model to be trained;

[0097] A second triple loss function is calculated based on the head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, and the model parameters of the second network model to be trained are reversely adjusted until the second triple loss function is minimized to obtain a trained second network model.

[0098] In this embodiment, the triple loss function is also used as the loss function of the second network model to perform model training; a first head posture image sample with a head posture label is randomly selected as the anchor sample x a , select and anchor sample x a The second head pose image sample with the same head pose label is used as the positive sample x p , the second head pose image sample and the first head pose image sample should belong to different objects; select and anchor sample x a The first head pose image sample with different head pose labels is used as the negative sample x n , that is, negative sample x n With anchor sample x a belong to the same object but have different head pose labels; the anchor sample, positive sample and negative sample are composed of three head pose image sample pairs (x a ,x p ,xn ) is fed into the second network model for training. For each sample, the second network model predicts and outputs the corresponding head pose feature vector. It then calculates a triple loss function based on the head pose feature vectors corresponding to the anchor sample, positive sample, and negative sample. The model parameters of the second network model are optimized until the triple loss function is minimized, indicating that the second network model training is complete.

[0099] In a preferred embodiment, a triple loss function is calculated based on the head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, specifically:

[0100] calculating a third similarity between the head pose feature vectors of the anchor sample and the positive sample, and a fourth similarity between the head pose feature vectors of the anchor sample and the negative sample;

[0101] A triple loss function is calculated according to the difference between the third similarity and the fourth similarity, and the triple loss function is expressed as follows:

[0102] L triplet (x n ,x a ,x p )=max(0,D(x a ,x n )-D(x a ,x p )+γ)

[0103]

[0104] Among them, (x n ,x a ,x p ) represents the three input head posture sample pairs, f(x a ) represents the head posture feature vector of the anchor sample extracted by the second network model, f(x n ) represents the head pose feature vector of the negative sample, f(x p ) represents the head posture feature vector of the positive sample;

[0105] D(x a ,x p ) represents the third similarity, D(x a ,x n ) represents the fourth similarity; γ represents the set second distance threshold.

[0106] By continuously improving the anchor sample x a and positive samples x p Similarity and anchor sample x a and negative samples x nThe difference between them is used to optimize the second network model, making the model more robust.

[0107] Multimodal fusion models also include aggregation networks and classification networks;

[0108] The aggregation network is connected to the output sides of the first network model and the second network model respectively, and is used to aggregate the voiceprint emotion features output by the first network model and the head posture features output by the second network model to obtain aggregated features;

[0109] In a specific example, the classification network may adopt a softmax classifier, which generates the current voiceprint emotional state and head posture angle of the object to be detected based on the aggregated features.

[0110] S3 generates a current concentration detection result of the subject to be detected based on the voiceprint emotional state and head posture angle;

[0111] After obtaining the voiceprint emotional state and head posture angle of the subject to be detected through the multimodal fusion model, different methods can be used to judge the learning concentration of the subject to be detected based on the voiceprint emotional state and head posture angle.

[0112] In a specific example, the voiceprint emotional state and head posture angle are input into a learning concentration detection model. The learning concentration detection model is a pre-trained model that calculates the current concentration state of the object to be detected based on the head posture angle and a preset learning space; and can obtain the current concentration detection result of the object to be detected based on preset standards and in combination with the voiceprint emotional state and concentration state.

[0113] For example, when the learning concentration detection model detects that the voiceprint emotion is not negative, it is judged as concentration, otherwise it is judged as non-concentration;

[0114] The horizontal distance y between the head of the object to be detected and the learning device, and the distance h to the left of the screen l 、Distance from the right side of the screen h r Together they determine the range of the learning interval (θ1, θ2):

[0115]

[0116] If the head posture angle is detected to be within the pre-set learning space, the learner is focused, otherwise he is not focused; the fusion state of the voiceprint emotional state and the head posture state is used as the learner's current concentration state.

[0117] See also Figure 4, judging the learner's learning concentration based on the voiceprint emotional state and head posture estimation data, specifically including: initializing the concentration score, setting the division thresholds of positive and negative states to 1 and 0 respectively, determining the head posture in the learning space as a positive state, i.e. 1, and determining the head posture outside the learning space as a negative state, i.e. 0, determining the negative state as 0 if the voiceprint emotion is negative, otherwise determining it as a positive state 1; calculating the sum of the head posture estimation score and the voiceprint emotion score as a fusion score; judging whether the fusion score is greater than or equal to 1, if it is, determining the current concentration as a positive state, i.e., concentration is concentrated, otherwise determining it as a negative state, i.e., lack of concentration.

[0118] After obtaining the concentration results of the subjects to be tested, corresponding intervention measures can be provided when the concentration is not focused, such as issuing voice warnings, to further focus the learners' concentration and improve their independent learning ability online.

[0119] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in this specific order, or that all of the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flowcharts can be performed in a different order. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or a step can be broken down into multiple steps.

[0120] This embodiment also provides a computer device, such as Figure 5 As shown, it includes at least one processor and at least one memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the above-mentioned learning concentration detection method. The specific steps are shown in the previous embodiment and will not be repeated here. In this embodiment, the types of processors and memories are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc. The memory can be a volatile memory, a non-volatile memory, or a combination thereof.

[0121] In addition, the computer device may also communicate with one or more external devices (such as a keyboard, a pointing terminal, a display, etc.), one or more terminals that enable a user to interact with the computer device, and / or any terminal that enables the computer device to communicate with one or more other computing terminals (such as a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the computer device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter.

[0122] It should be noted that the computer device can be an external monitoring device independent of the learning device, or it can refer to the learning device. In this case, a computer program that can execute the above-mentioned learning concentration detection method needs to be embedded in the learning device. The computer device preferably has a voice broadcast function, which can provide voice prompts when it detects that the learner is not concentrating.

[0123] This embodiment also provides a computer-readable medium storing a computer program executable by a computer device. When the computer program is executed on the computer device, the computer device executes the steps of the above-described learning concentration detection method. Types of computer-readable media include, but are not limited to, SD cards, USB flash drives, fixed hard drives, removable hard drives, and other storage media.

[0124] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting learning concentration, characterized in that: include: Acquire the voiceprint signal and head posture image of the object to be detected; Input the voiceprint signal and head posture image into the trained multimodal fusion model to extract the current voiceprint emotion characteristics and head posture characteristics of the object to be detected; Among them, the multimodal fusion model includes a first network model and a second network model; the first network model is used to extract the emotional features of the voiceprint signal, including a first feature extraction layer, a second feature extraction layer and a third feature extraction layer, and an attention mechanism module is set between the second feature extraction layer and the third feature extraction layer; the first feature extraction layer includes a one-dimensional convolution layer, a batch normalization processing layer and a ReLu layer; the second feature extraction layer includes multiple stacked feature extractors, which further extract features from the voiceprint feature representation output by the first feature extraction layer; the third feature extraction layer is implemented based on the ResNet34 network, and its function is to extract the emotional feature representation of the voiceprint signal of the object to be detected based on the voiceprint features output by the second feature extraction layer; the first network model is trained using at least one audio sample with an emotional label, and the second network model is trained using at least one head posture image sample with a head posture label; the multimodal fusion model fuses and classifies the voiceprint emotional features output by the first network model and the head posture features output by the second network model to obtain the current voiceprint emotional state and head posture angle of the object to be detected; The multimodal fusion model also includes an aggregation network and a classification network; the aggregation network is used to aggregate the voiceprint emotion features output by the first network model and the head posture features output by the second network model to obtain an aggregated feature; the classification network generates the current voiceprint emotion state and head posture angle of the object to be detected based on the aggregated feature; The voiceprint emotional state and head posture angle are input into the learning concentration detection model, and the current concentration state of the object to be detected is calculated according to the head posture angle and the preset learning space; the voiceprint emotional state and concentration state are combined to generate the current concentration detection result of the object to be detected.

2. The method for detecting learning concentration according to claim 1, wherein: The training process of the first network model includes: Obtaining a first audio sample with a first emotion label as an anchor sample, a second audio sample with the first emotion label as a positive sample, and a first audio sample with a second emotion label as a negative sample; wherein the second emotion label is different from the first emotion label, and the second audio sample and the first audio sample belong to different objects; Inputting the anchor sample, positive sample and negative sample into the first network model, and generating emotion feature vectors corresponding to the anchor sample, positive sample and negative sample respectively through the first network model to be trained; A first triple loss function is calculated based on the emotion feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, and the model parameters of the first network model to be trained are reversely adjusted until the first triple loss function is minimized to obtain a trained first network model.

3. The method for detecting learning concentration according to claim 2, wherein: The first triple loss function is calculated based on the emotion feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, specifically: Calculating a first similarity between the emotion feature vectors of the anchor sample and the positive sample, and a second similarity between the emotion feature vectors of the anchor sample and the negative sample; A first triple loss function is calculated according to the difference between the first similarity and the second similarity, where the first triple loss function is expressed as follows: in, represents the emotional feature vector of the anchor sample extracted by the first network model, represents the sentiment feature vector of the positive sample, The sentiment feature vector representing the negative sample; represents the first similarity, represents the second similarity, α is the first distance threshold set during network optimization; n c Indicates the number of training times.

4. The method for detecting learning concentration according to claim 1, wherein: The training process of the second network model includes: Acquire a first head pose image sample with a first head pose label as an anchor sample, a second head pose image sample with the first head pose label as a positive sample, and a first head pose image sample with a second head pose label as a negative sample; wherein the second head pose label is different from the first head pose label, and the second head pose image sample and the first head pose image sample belong to different objects; Inputting the anchor sample, the positive sample, and the negative sample into the second network model, and generating head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample respectively through the second network model to be trained; A second triple loss function is calculated based on the head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, and the model parameters of the second network model to be trained are reversely adjusted until the second triple loss function is minimized to obtain a trained second network model.

5. The method for detecting learning concentration according to claim 4, wherein: The second triple loss function is calculated based on the head posture feature vectors corresponding to the anchor sample, the positive sample, and the negative sample, specifically: calculating a third similarity between the head pose feature vectors of the anchor sample and the positive sample, and a fourth similarity between the head pose feature vectors of the anchor sample and the negative sample; A second triple loss function is calculated according to the difference between the third similarity and the fourth similarity, where the second triple loss function is expressed as follows: L triplet (x n ,x a ,x p )=max(0,D(x a ,x n )-D(x a ,x p )+γ) Among them, (x n ,x a ,x p ) represents three head pose sample pairs, corresponding to negative samples, anchor samples and positive samples respectively; f(x a ) represents the head posture feature vector of the anchor sample extracted by the second network model, f(x n ) represents the head pose feature vector of the negative sample, f(x p ) represents the head posture feature vector of the positive sample; D(x a ,x p ) represents the third similarity, D(x a ,x n ) represents the fourth similarity; γ represents the set second distance threshold.

6. The method for detecting learning concentration according to claim 1 or 2, wherein: Before inputting the voiceprint signal and the head posture image into the trained multimodal fusion model, the method further includes: Pre-processing the voiceprint signal using pre-emphasis, framing and windowing operations, and / or, The bicubic interpolation algorithm is used to perform image scaling on the head posture image.

7. A computer device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 6.

8. A computer-readable medium, characterized in that It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 6.

Images