A method and system for generating student behavior log based on audio scene recognition

Abstract

The present invention belongs to the technical field of audio scene recognition, and provides a method and system for generating a student behavior log based on audio scene recognition. The method includes obtaining a recorded audio document, dividing the audio document into a plurality of audio segments; using a trained educational scene recognition model based on the audio segments to obtain educational scene prediction labels for each audio segment; marking the start time point and the end time point for each audio segment, and generating a time distribution behavior log based on the marked start time point and the end time point of each audio segment combined with the educational scene prediction labels of each audio segment; wherein the process of prediction by the educational scene recognition model includes extracting the spectral features of the audio document and the probability distribution of the theme on the audio words, generating the theme features based on the probability distribution of the theme on the audio words and the spectral features of the audio document, updating the theme features, and classifying based on the updated theme features to obtain the educational scene prediction labels.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio scene recognition, and in particular relates to a method and system for generating a student behavior log based on audio scene recognition. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Education in the "Internet +" era is a flexible, diverse, open and personalized education. This kind of education emphasizes the individuality and diversity of learners, and its educational form is characterized by heterogeneity, dynamic change, openness and sharing. This new form of education makes the educational scene no longer limited to the traditional classroom education scene, but gradually transforms to a diversified educational scene. This change in educational scene has an urgent need for educational scene recognition technology. If the scene recognition technology can be used to identify the scenes where students are in different time periods of the day, a student behavior log can be generated based on the recognition results. The generated behavior log can be used for daily behavior analysis of students, helping parents to grasp the dynamics of students in real time, helping educational institutions to understand students' interests and hobbies, and providing students with accurate personalized learning services.

[0004] At present, mobile communication devices such as mobile phones and phone watches have become very popular among students, making it possible to record scenes in students' daily lives. Scenes can be recorded in the form of video and audio, but video data storage is large, the recording process consumes a lot of power, and is easily affected by factors such as lighting and occlusion. Therefore, recording scenes in the form of video has high requirements for mobile communication devices and the external environment. Due to the limitations of mobile communication equipment standby time, storage capacity and other conditions, it is difficult to record scenes through video. Compared with video, audio has the advantages of small data volume and is not easily affected by light occlusion, so it is easier to record scenes in the form of audio.

[0005] In summary, the demand for educational scene recognition technology due to the current changes in educational scenes can be met through audio scene recognition technology based on audio data. At present, there is no report on the research of generating student behavior logs through audio scene recognition technology. Summary of the invention

[0006] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a method and system for generating a student behavior log based on audio scene recognition. The method analyzes and identifies the recorded audio data of students' daily life through the scene recognition method, and can accurately identify the scenes in which students are located in different time periods, thereby generating a detailed and reliable student behavior log.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides a method for generating a student behavior log based on audio scene recognition.

[0009] A method for generating a student behavior log based on audio scene recognition, comprising:

[0010] Get the recorded audio file and divide the audio file into several audio segments;

[0011] Based on the audio clips, a trained educational scene recognition model is used to obtain educational scene prediction labels for each audio clip;

[0012] Mark the start time point and end time point for each audio clip, and generate a time distribution behavior log based on the marked start time point and end time point of each audio clip and the educational scenario prediction label of each audio clip;

[0013] Among them, the prediction process of the educational scene recognition model includes extracting the spectral features of the audio document and the probability distribution of the theme on the audio words, generating topic features based on the probability distribution of the theme on the audio words and the spectral features of the audio document, updating the topic features, and classifying based on the updated topic features to obtain the educational scene prediction label.

[0014] A second aspect of the present invention provides a student behavior log generation system based on audio scene recognition.

[0015] A student behavior log generation system based on audio scene recognition, comprising:

[0016] An acquisition module is configured to: acquire a recorded audio document and divide the audio document into a plurality of audio segments;

[0017] A prediction module is configured to: obtain an educational scene prediction label of each audio segment using a trained educational scene recognition model based on the audio segment;

[0018] A behavior log generation module is configured to: mark a start time point and an end time point for each audio segment, and generate a time distribution behavior log according to the marked start time point and end time point of each audio segment combined with the educational scenario prediction label of each audio segment;

[0019] Among them, the prediction process of the educational scene recognition model includes: a feature generation module, which is configured to extract the spectral features of the audio document and the probability distribution of the theme on the audio words; a topic decoupling module, which is configured to generate topic features based on the probability distribution of the theme on the audio words and the spectral features of the audio document; a topic interaction module, which is configured to update the topic features; and a classification output module, which is configured to perform classification based on the updated topic features to obtain the educational scene prediction label.

[0020] A third aspect of the present invention provides a computer-readable storage medium.

[0021] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the method for generating a student behavior log based on audio scene recognition as described in the first aspect above.

[0022] A fourth aspect of the present invention provides a computer device.

[0023] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for generating a student behavior log based on audio scene recognition as described in the first aspect above are implemented.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention innovatively introduces audio scene recognition technology into the field of educational artificial intelligence, and pioneeringly uses audio scene recognition technology to generate student behavior logs. The generated behavior logs can be used for daily behavior analysis of students, helping parents to understand students' dynamics in real time, helping educational institutions to understand students' interests and hobbies, and providing students with accurate personalized learning services.

[0026] 2. The features of long-term audio vary greatly at different time points, and traditional audio feature extraction methods cannot effectively extract long-term audio information. This invention innovatively decouples the topic features from long-term audio based on the topic model and attention mechanism. The decoupled topic features integrate the contextual information of long-term audio and can fully express the features of long-term audio.

[0027] 3. The present invention innovatively realizes the semantic interaction of topic features through graph neural networks, and utilizes the correlation between topic features to enhance the semantic expression of each topic feature. The updated topic features have stronger category distinction capabilities and can achieve more accurate recognition and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 is a structural block diagram of a student behavior log generation system based on audio scene recognition shown in the present invention;

[0030] Figure 2 It is a schematic diagram of the subject decoupling module shown in the present invention;

[0031] Figure 3 is a schematic diagram of a subject interaction module shown in the present invention;

[0032] Figure 4 It is a schematic diagram of the classification output module shown in the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0036] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0037] Embodiment 1

[0038] The present embodiment provides a method for generating a student behavior log based on audio scene recognition. The present embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal, a server, and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In the present embodiment, the method includes the following steps:

[0039] Get the recorded audio file and divide the audio file into several audio segments;

[0040] Based on the audio clips, a trained educational scene recognition model is used to obtain educational scene prediction labels for each audio clip;

[0041] Mark the start time point and end time point for each audio clip, and generate a time distribution behavior log based on the marked start time point and end time point of each audio clip and the educational scenario prediction label of each audio clip;

[0042] Among them, the prediction process of the educational scene recognition model includes extracting the spectral features of the audio document and the probability distribution of the theme on the audio words, generating topic features based on the probability distribution of the theme on the audio words and the spectral features of the audio document, updating the topic features, and classifying based on the updated topic features to obtain the educational scene prediction label.

[0043] As one or more implementation modes, the process of training the educational scene recognition model includes:

[0044] Extract the spectral features of the audio document and the probability distribution of the topic on the audio words;

[0045] Generate a topic feature matrix based on the probability distribution of topics on audio words and the spectral features of audio documents;

[0046] Update the topic feature matrix based on graph neural network;

[0047] Classify the updated topic feature matrix;

[0048] Output audio scene recognition results.

[0049] As one or more implementation modes, the process of extracting the spectral features of the audio document and the probability distribution of the topic on the audio words includes:

[0050] Label each audio document of each type of educational scene with an educational scene label, and construct a training set based on the audio document feature matrix and educational labels of all types of educational scenes;

[0051] Perform K-means clustering on all audio frames in the training set to obtain audio words;

[0052] For each audio document in the training set, count the number of occurrences of each audio word in each audio document, generate a histogram of audio words for each audio document, and obtain an "audio document-audio word" co-occurrence matrix;

[0053] The topic model "audio document-audio word" co-occurrence matrix is ​​used for decomposition to obtain the "topic-audio word" matrix and the "audio document-topic" matrix;

[0054] Based on the "topic-audio word" matrix and the "audio document-topic" matrix, the probability distribution of topics on audio words is obtained.

[0055] As one or more implementation modes, the process of generating topic features based on the probability distribution of topics on audio words and the spectral features of the audio document includes:

[0056] Using a first fully connected layer to map the frequency spectrum features of the audio document to obtain a mapped feature vector matrix;

[0057] Using a second fully connected layer to map the probability distribution of the topic on the audio words to obtain a mapped probability distribution matrix;

[0058] Based on the mapped feature vector matrix and the mapped probability distribution matrix, an attention model is used to construct a topic feature matrix.

[0059] As one or more implementation modes, the process of updating the topic features based on the graph neural network includes:

[0060] Based on the topic feature matrix, the graph attention network is used to update each feature of the topic feature matrix to obtain the updated topic feature matrix.

[0061] As one or more implementation modes, the process of classifying the updated subject features includes:

[0062] Based on the updated topic feature matrix, a third fully connected layer mapping is used to obtain a probability distribution of each audio document in each type of educational scenario;

[0063] Based on the probability distribution of each audio document in each type of educational scene, the educational scene prediction label of the audio document is obtained.

[0064] As one or more implementation modes, the process of generating a time-distributed behavior log includes:

[0065] The educational scene prediction labels are based on the set order of the audio clips, and adjacent audio clips with the same prediction labels are synthesized into a whole clip to obtain several integrated audio clips of different lengths. The educational scene labels, starting time points, and ending time points of each integrated audio clip are recorded to generate a time distribution behavior log.

[0066] Embodiment 2

[0067] This embodiment provides a student behavior log generation system based on audio scene recognition.

[0068] A student behavior log generation system based on audio scene recognition, comprising:

[0069] An acquisition module is configured to: acquire a recorded audio document and divide the audio document into a plurality of audio segments;

[0070] A prediction module is configured to: obtain an educational scene prediction label of each audio segment using a trained educational scene recognition model based on the audio segment;

[0071] A behavior log generation module is configured to: mark a start time point and an end time point for each audio segment, and generate a time distribution behavior log according to the marked start time point and end time point of each audio segment combined with the educational scenario prediction label of each audio segment;

[0072] Among them, the prediction process of the educational scene recognition model includes: a feature generation module, which is configured to extract the spectral features of the audio document and the probability distribution of the theme on the audio words; a topic decoupling module, which is configured to generate topic features based on the probability distribution of the theme on the audio words and the spectral features of the audio document; a topic interaction module, which is configured to update the topic features; and a classification output module, which is configured to perform classification based on the updated topic features to obtain the educational scene prediction label.

[0073] like Figure 1As shown, this embodiment consists of the following five modules: feature generation module, topic decoupling module, topic interaction module, classification output module and behavior log generation module, among which the topic decoupling module, topic interaction module and classification output module constitute the educational scene recognition model. The function of the feature generation module is to extract the spectral features of the audio document and the probability distribution of the topic on the audio words; the topic decoupling module generates topic features based on the probability distribution of the topic on the audio words and the spectral features of the audio document; the topic interaction module updates the topic features based on the graph neural network; the classification output module performs classification based on the updated topic features and outputs the audio scene recognition results.

[0074] The whole process of this embodiment includes two stages: the training stage of the educational scene recognition model and the generation stage of the student behavior log. The training stage of the educational scene recognition model is composed of a feature generation module, a theme decoupling module, a theme interaction module, and a classification output module. The student behavior log generation stage will use the trained educational scene recognition model and the behavior log generation module to generate the student behavior log, so all modules of the feature generation module, theme decoupling module, theme interaction module, classification output module and behavior log generation module are needed.

[0075] This embodiment will introduce the student behavior log method proposed in this embodiment according to the education recognition model training stage and the student behavior log generation stage.

[0076] ⅠEducational scene recognition model training stage

[0077] 1. Feature Generation Module

[0078] (1) Collect audio data of M types of educational scenes, and collect N audio files of 10 seconds in length for each type of educational scene.

[0079] (2) Each audio clip of each type of audio scene is divided into frames, assuming that each audio document is divided into R frames. The 128-dimensional log-mel features commonly used in the audio field are extracted for each audio frame. The audio features of all frames in each audio document constitute the feature matrix of the audio document, and a log-mel feature matrix of size 128×R is obtained for each audio document. An educational scene label is marked for each audio document of each type of educational scene. The audio document feature matrix and educational labels of all types of educational scenes constitute the training set of the educational scene recognition model, denoted as Among them, X m,n The log-mel feature matrix representing the nth audio document of the mth educational scene, Represents the feature matrix X m,n The rth column of Y is the feature vector of the rth frame of the audio document, r = 1,…,R. m,nThe educational scene label Y represents the educational scene label of the nth audio document of the mth educational scene. m,n It is represented in the form of a one-hot vector.

[0080] (3) All audio frames in the audio feature training set are clustered using K-means. Assume that K clusters are obtained by clustering, and the cluster center of each cluster is denoted as w k ,k=1,…,K. The cluster center w k As audio words, a total of K different audio words are obtained, and the K audio words form an audio dictionary Dic={w k} k＝1,…,K .

[0081] (4) Map each audio frame in the training set using the following formula to obtain the corresponding audio word index:

[0082]

[0083] in, Represents an audio frame The audio word index obtained after mapping, Represents an audio frame With audio word w k For each audio document in the training set, the number of occurrences of each audio word in each audio document is counted, and a histogram of audio words is generated for each audio document. The audio word histograms of all audio documents in the training set form the "audio document-audio word" co-occurrence matrix H.

[0084]

[0085] in, represents the number of occurrences of the kth audio word in the lth audio document in the training set, k = 1, ..., K, l = 1, ..., L, K represents the number of audio words in the audio dictionary, L represents the number of audio documents in the training set, L = MN. The LDA (Latent Dirichlet Allocation) topic model is used to perform matrix decomposition on H to obtain the "topic-audio word" matrix Θ and the "audio document-topic" matrix Φ.

[0086]

[0087] in, is the k-th row and t-th column element of the Θ matrix, indicating the number of occurrences of the k-th audio word in the t-th topic, T represents the number of topics, K represents the number of audio words, represents the probability of occurrence of the tth topic in the lth audio document,

[0088] (5) The output of the feature generation module is the training set and the “topic-audio word” matrix Θ.

[0089] 2. Theme Decoupling Module

[0090] (1) The topic decoupling module uses the training set and the “topic-audio word” matrix Θ as input.

[0091] (2) Figure 2 As shown, the topic decoupling module has two inputs, which are all feature matrices X in the training set. m,n They are used as the input of the fully connected layer 1 and Θ as the input of the fully connected layer 2.

[0092] The fully connected layer 1 converts X m,n Each frame is mapped as shown in the following formula to obtain the mapped feature vector

[0093]

[0094] Where W1 represents the weight matrix of the fully connected layer 1, and its value is obtained through training of the educational scene recognition model. ReLU() represents the activation function of the fully connected layer 1. r=1,…,R forms a matrix

[0095] The fully connected layer 2 converts all column vectors Θ in Θ t Perform the mapping shown below to obtain

[0096]

[0097] Among them, W2 represents the weight matrix of the fully connected layer 2, the matrix size is K×K, and its value is obtained through the training of the educational scene recognition model. Softmax() represents the activation function of the fully connected layer 2. t=1,…,T forms a matrix

[0098] (3) and As the input of the attention model, the topic features are obtained by performing topic decoupling through the following formula:

[0099]

[0100] Among them, F t m,n Represents the topic features of the tth topic of the nth audio document of the mth educational scene in the training set. express The index is The value of . The topic features of all topics are obtained by decoupling the attention model to form a topic feature matrix

[0101] 3. Theme Interaction Module

[0102] (1) Figure 3 As shown in the figure, the topic interaction module consists of B layers of graph attention network layers. The topic feature matrix F in the training set is m,n ,m=1,…,M,n=1,…,N are simultaneously sent to the subject interaction module, with F m,n T topic features F1 in m,n ,…,F T m,n As the node features of T nodes in the first layer of graph attention network, that is,

[0103]

[0104] in, Represents the node features of the tth node after the first layer of the graph attention network.

[0105] (2) The node features of other layers of graph attention networks are calculated as follows:

[0106]

[0107] in,

[0108]

[0109] h t b represents the t-th node feature of the b-th layer graph attention network, h j b represents the j-th node feature of the b-th layer graph attention network, h i b represents the i-th node feature of the b-th layer graph attention network, b=1,…,B,t=1,…,T,j=1,…,T. ELU() represents the activation function, W and V represent the weight matrix of the graph attention network, and their values ​​are obtained through training of the educational scene recognition model. LeakyReLU() represents the activation function. ‖ represents the concatenation operation of the vector.

[0110] (3) The T node features h calculated by the last layer of graph attention network are i B Perform the connection operation to obtain the output feature h B ,

[0111] 4. Classification Output Module

[0112] (1) Figure 4 As shown, h B As the input of the classification output module, after being mapped by the fully connected layer 3, the probability distribution of each audio document on M educational scene classes is obtained.

[0113] y m,n =softmax(W3h B )

[0114] Among them, y m,n It represents the probability distribution of the nth audio document of the mth educational scene class in the training set on M educational scene classes. W3 represents the weight matrix of the fully connected layer 3, and its value is obtained through training of the educational scene recognition model.

[0115] (2) The recognition result of the audio document is obtained by the following formula:

[0116]

[0117] Among them, Label(X m,n ) indicates that X m,n The predicted class label, whose value is in {1,…,M}. m,n (i) represents y m,n The number of class labels is the number of educational scene classes.

[0118] The training phase of the educational scene recognition model consists of the above four modules, with class cross entropy as the loss function, and training is performed through iteration. After the training is completed, the model parameters are fixed to obtain the trained educational scene recognition model. L = -∑Y m, n logy m,n

[0119] (3) Save the trained educational scene recognition model.

[0120] Ⅱ Student Behavior Log Generation Phase

[0121] 1. Feature Generation Module

[0122] The recorded audio file is divided into S 10-second audio clips. Each audio clip is sent to the optimal educational scene recognition model, and the frame division and feature extraction are performed according to the steps in the training phase to obtain the feature matrix Z s , s=1,…,S.

[0123] 2. Theme Decoupling Module

[0124] The "topic-audio word" matrix Θ and feature matrix Z obtained in the training phase of the educational scene recognition model are s Send it to the topic decoupling module trained in the training phase to obtain the topic feature, denoted as ZF s .

[0125] 3. Theme Interaction Module

[0126] The subject feature ZF s As the input of the topic interaction module, it is sent to the topic interaction module trained in the training phase to obtain the output feature vector h s B , s=1,…,S.

[0127] 4. Classification Output Module

[0128] The output feature vector h s B As the input of the classification output module, it is sent to the classification output module trained in the training phase to obtain the educational scene prediction label Label (Z s ).

[0129] 5. Behavior log generation module

[0130] (1) Mark the start and end time points for each audio segment (ST s ,ET s ),s=1,…,S. ST s Indicates the starting time of the sth audio segment, ET s Indicates the end time of the sth audio segment

[0131] (2) Mark the start and end time points of each audio segment (ST s ,ET s ) and educational scene identification label Label(Z s )∈{1,…,M}, the generated statistical behavior log RATE m The proportion of each educational scene in the entire audio document is calculated as shown in the following formula:

[0132] RATE m =NUM m / S

[0133] Among them, RATE m Indicates the proportion of the mth type of educational scene in the entire audio document, NUM m Represents the number of audio clips in the audio document whose predicted label is the mth type of educational scene.

[0134] (3) Mark the start and end time points of each audio segment (ST s,ET s ) and educational scenario prediction label Label(Z s )∈{1,…,M}, the generated time distribution behavior log. s ), s=1,…,S, scan the adjacent audio clips with the same predicted label from small to large according to s to synthesize an overall clip, obtain C audio clips of different lengths after integration, record the educational scene labels and start and end time points of each integrated audio clip, and generate a time distribution behavior log, as shown in the following formula

[0135] {(ST c ,ET c ):Label c |c=1,2,…,C}

[0136] Among them, ST c Indicates the starting time point of the cth audio segment after integration, ET c Indicates the end time point of the cth audio segment after integration, Label c Represents the educational scene label of the c-th audio segment after integration.

[0137] Embodiment 3

[0138] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the method for generating a student behavior log based on audio scene recognition as described in the first embodiment above are implemented.

[0139] Embodiment 4

[0140] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for generating a student behavior log based on audio scene recognition as described in the first embodiment above are implemented.

[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating student behavior logs based on audio scene recognition, characterized in that: include: Get the recorded audio file and divide the audio file into several audio segments; Based on the audio clips, a trained educational scene recognition model is used to obtain educational scene prediction labels for each audio clip; Mark the start time point and end time point for each audio clip, and generate a time distribution behavior log based on the marked start time point and end time point of each audio clip and the educational scenario prediction label of each audio clip; The process of predicting the educational scene recognition model includes extracting the spectral features of the audio document and the probability distribution of the topic on the audio words, generating topic features based on the probability distribution of the topic on the audio words and the spectral features of the audio document, updating the topic features, and performing classification based on the updated topic features to obtain the educational scene prediction label; The process of training the educational scene recognition model includes: Extract the spectral features of the audio document and the probability distribution of the topic on the audio words; Generate a topic feature matrix based on the probability distribution of topics on audio words and the spectral features of audio documents; Update the topic feature matrix based on graph neural network; Classify the updated topic feature matrix; Output audio scene recognition results; The process of updating the topic features based on the graph neural network includes: Based on the topic feature matrix, the graph attention network is used to update each feature of the topic feature matrix to obtain the updated topic feature matrix.

2. The method for generating a student behavior log based on audio scene recognition according to claim 1 is characterized in that: The process of extracting the frequency spectrum features of the audio document and the probability distribution of the topic on the audio word comprises: Label each audio document of each type of educational scene with an educational scene label, and construct a training set based on the audio document feature matrix and educational labels of all types of educational scenes; Perform K-means clustering on all audio frames in the training set to obtain audio words; For each audio document in the training set, count the number of occurrences of each audio word in each audio document, generate a histogram of audio words for each audio document, and obtain an "audio document-audio word" co-occurrence matrix; The topic model "audio document-audio word" co-occurrence matrix is ​​used for decomposition to obtain the "topic-audio word" matrix and the "audio document-topic" matrix; Based on the "topic-audio word" matrix and the "audio document-topic" matrix, the probability distribution of topics on audio words is obtained.

3. The method for generating a student behavior log based on audio scene recognition according to claim 1 is characterized in that: The process of generating topic features based on the probability distribution of topics on audio words and the spectrum features of audio documents includes: Using a first fully connected layer to map the frequency spectrum features of the audio document to obtain a mapped feature vector matrix; Using a second fully connected layer to map the probability distribution of the topic on the audio words to obtain a mapped probability distribution matrix; Based on the mapped feature vector matrix and the mapped probability distribution matrix, an attention model is used to construct a topic feature matrix.

4. The method for generating a student behavior log based on audio scene recognition according to claim 1 is characterized in that: The process of classifying the updated subject features includes: Based on the updated topic feature matrix, a third fully connected layer mapping is used to obtain a probability distribution of each audio document in each type of educational scenario; Based on the probability distribution of each audio document in each type of educational scene, the educational scene prediction label of the audio document is obtained.

5. The method for generating student behavior logs based on audio scene recognition according to claim 1 is characterized in that: The process of generating a time distribution behavior log includes: The educational scene prediction labels are based on the set order of the audio clips, and adjacent audio clips with the same prediction labels are synthesized into a whole clip to obtain several integrated audio clips of different lengths. The educational scene labels, starting time points, and ending time points of each integrated audio clip are recorded to generate a time distribution behavior log.

6. A student behavior log generation system based on audio scene recognition, characterized in that: include: An acquisition module is configured to: acquire a recorded audio document and divide the audio document into a plurality of audio segments; A prediction module is configured to: obtain an educational scene prediction label of each audio segment using a trained educational scene recognition model based on the audio segment; A behavior log generation module is configured to: mark a start time point and an end time point for each audio segment, and generate a time distribution behavior log according to the marked start time point and end time point of each audio segment combined with the educational scenario prediction label of each audio segment; The process of the educational scene recognition model prediction includes: a feature generation module, which is configured to extract the spectral features of the audio document and the probability distribution of the theme on the audio word; a topic decoupling module, which is configured to generate topic features based on the probability distribution of the theme on the audio word and the spectral features of the audio document; a topic interaction module, which is configured to update the topic features; a classification output module, which is configured to classify based on the updated topic features to obtain the educational scene prediction label; The process of training the educational scene recognition model includes: Extract the spectral features of the audio document and the probability distribution of the topic on the audio words; Generate a topic feature matrix based on the probability distribution of topics on audio words and the spectral features of audio documents; Update the topic feature matrix based on graph neural network; Classify the updated topic feature matrix; Output audio scene recognition results; The process of updating the topic features based on the graph neural network includes: Based on the topic feature matrix, the graph attention network is used to update each feature of the topic feature matrix to obtain the updated topic feature matrix.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for generating a student behavior log based on audio scene recognition as described in any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the method for generating a student behavior log based on audio scene recognition as described in any one of claims 1 to 5 are implemented.

Images