Teaching interactive answering positioning method, device, computer equipment and storage medium

Abstract

The present invention belongs to the field of artificial intelligence, and relates to a teaching interactive answering positioning method, device, computer equipment and storage medium. The method includes: obtaining a set of video frame sequences and a corresponding mixed audio file; identifying key frames in the set of video frame sequences, performing feature extraction, and performing short-time Fourier transform on the mixed audio, and then performing feature extraction on the audio; performing multimodal fusion to generate a spectrum mask; calculating the momentum information of continuous video frames and generating momentum embedding, and performing spectrum embedding processing according to the spectrum mask; inputting a momentum-aware Transformer module to generate sequence information; performing residual fusion to generate a spectrum mask; and generating a visual sound source localization map. Multimodal fusion improves positioning accuracy, momentum embedding and spectrum embedding enhance timing sensitivity, residual fusion optimizes feature integration, and a visual sound source localization map enhances intuitiveness, enhances the interactivity and fun of learning, and helps to improve teaching efficiency and quality.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a teaching interactive answering positioning method, device, computer equipment and storage medium. Background Art

[0002] In the existing classroom interactive teaching environment, the positioning method of students' quick answers has inaccurate problems in some aspects, which is mainly reflected in: there are deficiencies in the technical implementation level. Most of the current quick answer systems rely on electronic devices, such as quick answer devices or online platforms, but the positioning accuracy and response speed of these devices may be affected by various factors such as the network and environment, resulting in the system being unable to accurately determine which student will answer first. Inaccurate quick answer positioning will weaken the fairness of the quick answer and affect students' learning enthusiasm. Inaccurate quick answer positioning will also affect teachers' teaching judgments, making it difficult for teachers to accurately grasp students' learning situations; it will also aggravate the chaos of classroom order and make it difficult for teaching to proceed smoothly. Current quick answer systems such as quick answer devices or online platforms are not conducive to the maintenance of the teaching environment and greatly reduce user satisfaction. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a teaching interactive answering positioning method, which adopts the following technical solution, including:

[0004] Acquire input data, the input data comprising a set of video frame sequences and a corresponding mixed audio file;

[0005] Identifying key frames in the set of the video frame sequence, performing feature extraction on the key frames to obtain first modal information, performing short-time Fourier transform on the mixed audio, and then performing feature extraction on the audio to obtain second modal information;

[0006] Performing multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result, and generating different spectrum masks;

[0007] Calculating momentum information of consecutive video frames and generating momentum embedding, and performing spectrum embedding processing according to the spectrum mask;

[0008] The momentum embedding and spectrum embedding are respectively input into the momentum-aware Transformer module to generate sequence information;

[0009] Performing residual fusion on the sequence information and the feature fusion result to generate a spectrum mask;

[0010] Audio separation is performed according to the spectrum mask, and a visual sound source localization map is generated through CAM.

[0011] In order to solve the above technical problems, the present invention also provides a teaching interactive answering positioning device, which adopts the following technical solution, including:

[0012] An acquisition module, used to acquire input data, wherein the input data includes a set of video frame sequences and a corresponding mixed audio file;

[0013] A feature extraction module, used to identify key frames in the set of the video frame sequence, perform feature extraction on the key frames to obtain first modal information, perform short-time Fourier transform on the mixed audio, and then perform feature extraction on the audio to obtain second modal information;

[0014] A multimodal fusion module, configured to perform multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result and generate different spectrum masks;

[0015] A processing module, used for calculating momentum information of consecutive video frames and generating momentum embedding, and performing spectrum embedding processing according to the spectrum mask;

[0016] A generation module, which is used to input momentum embedding and spectrum embedding into the momentum-aware Transformer module to generate sequence information;

[0017] A residual fusion module, used for performing residual fusion on the sequence information and the feature fusion result to generate a spectrum mask;

[0018] The localization module is used to perform audio separation according to the spectrum mask and generate a visual sound source localization map through CAM.

[0019] In order to solve the above technical problems, the present invention also provides a computer device, which adopts the technical solution described below, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above-mentioned teaching interactive answering positioning method are implemented.

[0020] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, which adopts the technical solution described below, and stores computer-readable instructions on the computer-readable storage medium. When the computer-readable instructions are executed by the processor, the steps of the above-mentioned teaching interactive answering positioning method are implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects: multimodal fusion improves positioning accuracy, momentum embedding and spectrum embedding enhance timing sensitivity, residual fusion optimizes feature integration, and the visualization of sound source localization map enhances intuitiveness, enhances the interactivity and fun of learning, and helps to improve teaching efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 It is a flow chart of an embodiment of the teaching interactive answering positioning method of the present invention;

[0024] Figure 2 It is a structural schematic diagram of an embodiment of the teaching interactive answering positioning device of the present invention;

[0025] Figure 3 It is a schematic structural diagram of an embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0029] It should be noted that the teaching interactive answering positioning method provided in the embodiment of the present invention is generally executed by a server / terminal device, and accordingly, the teaching interactive answering positioning device is generally arranged in the server / terminal device.

[0030] It should be understood that the number of terminal devices, networks and servers is only illustrative. Depending on the implementation requirements, there can be any number of terminal devices, networks and servers.

[0031] Embodiment 1

[0032] Please refer to Figure 1 , shows a flow chart of an embodiment of the teaching interactive answering positioning method of the present invention. The teaching interactive answering positioning method comprises the following steps:

[0033] Step S1, obtaining input data, the input data including a set of video frame sequences and corresponding mixed audio files.

[0034] In this embodiment, the electronic device (e.g., server / terminal device) on which the teaching interactive answering positioning method is running can receive the teaching interactive answering positioning request through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAXX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0035] In this embodiment, step S1, obtaining input data, the input data including a set of video frame sequences and a corresponding mixed audio file, may further include the following steps:

[0036] S11. Collect video frame sequences.

[0037] The teaching activities are recorded as video files by using a camera or video recording device. This step ensures the video data required for subsequent analysis.

[0038] Decode recorded video files into a sequence of frames. Video decoding is the process of converting a video file from a compressed format into a processable sequence of frames.

[0039] S12. Record the audio of the teaching activity site into a mixed audio file.

[0040] The audio of the teaching activity is recorded as a mixed audio file through a microphone or audio recording device. The mixed audio file contains the voices of all participants and is important data required for subsequent analysis.

[0041] Preprocess the recorded mixed audio files, including denoising, gain adjustment, and filtering. The preprocessing step helps improve the audio quality and provide accurate data for subsequent analysis.

[0042] Split the mixed audio file into audio segments corresponding to the video frame sequence. The purpose of audio segmentation is to synchronize the audio data with the video frame sequence to facilitate subsequent analysis.

[0043] Step S2, identifying key frames in the set of video frame sequences, performing feature extraction on the key frames to obtain first modal information, performing short-time Fourier transform on the mixed audio, and then performing feature extraction on the audio to obtain second modal information.

[0044] In this embodiment, step S2, identifying key frames in a set of video frame sequences, performing feature extraction on the key frames to obtain first modal information, performing short-time Fourier transform on the mixed audio, and then performing feature extraction on the audio to obtain second modal information may also specifically include the following steps:

[0045] S21. Identify key frames in a set of video frame sequences.

[0046] A key frame is a specific frame used to define changes in an animation or video. It marks the state value of a certain attribute at a specific point in time. In video processing, identifying key frames is the basis for video analysis, editing, and retrieval.

[0047] S211, read the video file.

[0048] First, you need to read the video file and load it into memory for subsequent processing. For example, you can use a video processing library (such as OpenCV) to read the video file and convert it into a frame-by-frame image sequence.

[0049] S212, pre-processing the read video file.

[0050] Preprocess the video frames to improve the accuracy of key frame recognition.

[0051] The read video files are preprocessed, including grayscale, denoising, image enhancement and other operations. Grayscale converts color images into grayscale images to reduce the amount of calculation; denoising removes noise interference in the image; image enhancement improves the contrast and clarity of the image.

[0052] S213, detecting key frames.

[0053] Identify key frames in a video, which are frames that contain important visual information or noticeable changes.

[0054] Methods based on motion detection, image difference, scene change, etc. may be used. For example, by calculating the image difference between adjacent frames, when the difference exceeds a certain threshold, the frame is considered to be a key frame.

[0055] S214, extract key frames.

[0056] Key frames are extracted from the video frame sequence to form a key frame set.

[0057] According to the results of key frame detection, key frames are filtered out from the video frame sequence and saved into the specified data structure.

[0058] S22, extracting features from the key frame to obtain first modal information.

[0059] Feature extraction is the process of extracting information that reflects its essential properties from raw data. For key frames, feature extraction can help identify and understand the content and structure in the video.

[0060] S221, select features.

[0061] Determine the type of features to be extracted that can reflect the visual information of the keyframe.

[0062] The features to be extracted include color features (such as color histogram), texture features (such as gray-level co-occurrence matrix), shape features (such as edge detection), etc.

[0063] S222, select a feature extraction algorithm.

[0064] For example, the SIFT (Scale Invariant Feature Transform) algorithm is used to extract local feature points of key frames; the HOG (Histogram of Oriented Gradients) algorithm is used to extract shape features of key frames.

[0065] S223, construct a feature vector.

[0066] The extracted features are converted into feature vectors for subsequent processing and analysis. The information of each feature point (such as position, direction, scale, etc.) is encoded into a feature vector, and all feature vectors are combined into a feature matrix.

[0067] S224, normalize the features.

[0068] The feature vector is normalized to improve the accuracy of subsequent classification and recognition.

[0069] The normalization methods that can be used include minimum-maximum normalization, Z-score normalization, etc.

[0070] S23, performing short-time Fourier transform (STFT) on the mixed audio, and then extracting features from the audio to obtain second modal information.

[0071] Audio feature extraction is a key technology in the field of speech recognition and natural language processing. Short-time Fourier transform (STFT) is a commonly used audio signal processing technology that can convert audio signals from the time domain to the frequency domain, thereby extracting information reflecting the audio features.

[0072] S231, pre-processing the audio.

[0073] Preprocess the original audio signal to improve the accuracy of subsequent processing.

[0074] Preprocess the audio, including framing, windowing, pre-emphasis, etc. Framing is to divide the audio signal into multiple short time periods; windowing is to smooth each short time period; pre-emphasis is to increase the amplitude of high-frequency components.

[0075] S232, performing short-time Fourier transform (STFT) on the preprocessed audio.

[0076] Convert the audio signal from time domain to frequency domain to obtain a spectrogram.

[0077] Perform Fourier transform on each short time period to obtain the spectrum of that time period; combine the spectra of all time periods to form a two-dimensional spectrum diagram (i.e., spectrogram).

[0078] S233, extracting features from the audio after short-time Fourier transformation.

[0079] The purpose of feature extraction of audio after short-time Fourier transform is to extract information reflecting the audio characteristics from the spectrum graph.

[0080] Audio features include spectral centroid, zero-crossing rate, MFCC (Mel-frequency cepstral coefficient), etc. Spectral centroid reflects the centroid position of the spectrum; zero-crossing rate indicates the number of times the audio signal crosses the zero axis; MFCC simulates the nonlinear perception characteristics of the human ear to frequency.

[0081] Extracting the spectral centroid is to calculate the spectral centroid of each frame in the spectrogram, reflecting the frequency distribution characteristics of the audio signal. The energy of each frequency band is weighted and summed to obtain the spectral centroid.

[0082] Extracting the zero-crossing rate is used to count the number of times the audio signal crosses the zero axis, reflecting the frequency components and rhythm characteristics of the audio signal. Traverse the sampling points of the audio signal and count the number of times it crosses the zero axis.

[0083] Extract MFCC, which is used to extract Mel frequency cepstral coefficients to reflect the speech characteristics of the audio signal.

[0084] The steps of extracting MFCC include: preprocessing, framing, windowing, pre-emphasis and other operations on the audio signal. Fast Fourier Transform (FFT), converting the audio signal from the time domain to the frequency domain. Mel filter bank, applying the Mel filter bank to smooth the spectrum and extract information reflecting the speech characteristics. Logarithmic operation and discrete cosine transform (DCT), logarithmic operation and discrete cosine transform (DCT) are performed on the output of the Mel filter bank to obtain the MFCC feature vector.

[0085] S233, feature vector construction and normalization.

[0086] The purpose of feature vector construction and normalization is to convert the extracted audio features into feature vectors and perform normalization processing.

[0087] The information of each audio feature point is encoded into a feature vector, and all feature vectors are combined into a feature matrix; the feature matrix is ​​normalized.

[0088] Step S3, performing multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result, and generating different spectrum masks.

[0089] In this embodiment, step S3, performing multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result, and generating different spectrum masks may specifically include the following steps:

[0090] In the teaching interactive answering positioning method, the first modal information (feature information of the video frame) and the second modal information (feature information of the audio) are multimodally fused to obtain a more comprehensive feature fusion result and generate different spectrum masks. This step is the key link to connect visual information and audio information, providing more accurate and rich input data for subsequent steps. Specifically, it can be divided into the following sub-steps:

[0091] S31, preprocessing the first modal information and the second modal information.

[0092] Before multimodal fusion, the first modal information and the second modal information need to be preprocessed to ensure that they can be effectively fused.

[0093] Video frame feature preprocessing: After extracting features from key frames, the feature vectors obtained may have different dimensions and distributions. In order to perform fusion, these feature vectors need to be standardized and normalized so that they have the same scale and distribution range.

[0094] Audio feature preprocessing: After performing short-time Fourier transform (STFT) on the mixed audio, the obtained spectral features also need to be preprocessed. This includes removing noise, enhancing signals, and other processing to improve the quality of audio features.

[0095] S32, aligning the video frame features and the audio features.

[0096] Since the sampling rates of video frames and audio are different, their features may not match in the time dimension. Therefore, before multimodal fusion, it is necessary to align the video frame features and audio features.

[0097] Temporal alignment: Align the video frame features and audio features in the temporal dimension by synchronizing the timestamps of the video frames and audio. This can be achieved through methods such as interpolation or resampling.

[0098] Spatial alignment: If the video frame features contain spatial information (such as position information in an image), this spatial information needs to be aligned with the audio features. This can be achieved by mapping the audio features to the spatial coordinates of the video frame.

[0099] S33, select feature fusion strategy.

[0100] When performing multimodal fusion, it is necessary to select an appropriate fusion strategy, which depends on the specific application scenario and data characteristics.

[0101] Early fusion: In early fusion, video frame features and audio features are combined in the feature extraction stage to form a unified feature vector. The advantage of this method is that it can make full use of the complementary information between the two modalities, but the disadvantage is that the computational complexity is high.

[0102] Late fusion: In late fusion, video frame features and audio features are processed separately and not fused until the final decision stage. The advantage of this approach is low computational complexity, but the disadvantage is that it may not fully utilize the complementary information between the two modalities.

[0103] Intermediate fusion: In intermediate fusion, video frame features and audio features are fused at an intermediate stage after feature extraction. The advantage of this method is that it can fully utilize the complementary information between the two modalities while maintaining low computational complexity.

[0104] In this embodiment, an intermediate fusion strategy is selected. Specifically, firstly, the video frame features and the audio features are subjected to dimensionality reduction processing respectively, and then they are fused together by methods such as splicing or weighted summation.

[0105] S34, fusing the video frame features and the audio features, and generating a feature fusion result.

[0106] Feature dimensionality reduction: In order to reduce computational complexity, the video frame features and audio features are first reduced in dimensionality, which can be achieved through methods such as principal component analysis (PCA) and linear discriminant analysis (LDA).

[0107] Feature splicing: Splice the reduced-dimensional video frame features and audio features to form a new feature vector. This new feature vector contains the joint information of the video frame and audio.

[0108] Weighted summation: Another fusion method is weighted summation. Different weights can be assigned to the video frame features and audio features according to their importance, and then the weighted sum is calculated as the fused feature vector to obtain the feature fusion result.

[0109] S35: Generate a spectrum mask.

[0110] After feature fusion is completed, different spectral masks need to be generated for audio separation and visual sound source localization in subsequent steps.

[0111] Spectral mask generation principle: Spectral mask is a signal processing technology used to separate different sound sources in mixed audio. It generates a mask signal that can suppress or enhance a specific sound source by performing spectral analysis on the mixed audio.

[0112] Based on the feature fusion results, a spectrum mask corresponding to the mixed audio is generated. This mask signal corresponds to the target sound source in the mixed audio in the spectrum and can be used to separate the target sound source in the subsequent steps.

[0113] Spectrum mask type: Depending on the application scenario and data characteristics, different types of spectrum masks can be generated, such as binary mask, soft mask, etc. These masks will play an important role in the subsequent steps.

[0114] Step S4, calculating momentum information of consecutive video frames and generating momentum embedding, and performing spectrum embedding processing according to the spectrum mask.

[0115] In this embodiment, step S4, calculating momentum information of continuous video frames and generating momentum embedding, and performing spectrum embedding processing according to the spectrum mask may further include the following steps:

[0116] S41, calculating momentum information of consecutive video frames.

[0117] In specific implementation, step S41, calculating the momentum information of consecutive video frames may include the following steps:

[0118] S411, extracting continuous video frames.

[0119] First, it is necessary to extract continuous video frames from the set of video frame sequences. This step is to analyze the temporal continuity and motion information between video frames. The extraction of continuous video frames can be achieved by reading the video file or video frame sequence frame by frame.

[0120] S412, calculating the difference between frames.

[0121] After extracting consecutive video frames, the difference between adjacent frames needs to be calculated. The difference between frames reflects the changes between video frames, which are usually caused by object motion, lighting changes, or camera jitter. The calculation of the difference between frames can be achieved by calculating the difference in pixel values ​​between adjacent frames, such as using absolute difference or square difference.

[0122] S413, calculating momentum information.

[0123] After obtaining the inter-frame difference, it is necessary to further calculate the momentum information. Momentum information refers to the rate of change of motion between video frames, reflecting the dynamic characteristics of motion in the video frame sequence. The calculation of momentum information can be achieved by integrating or differentiating the inter-frame difference in time. Specifically, the momentum information can be obtained by calculating the cumulative value or rate of change of the inter-frame difference.

[0124] S414, generating momentum embedding.

[0125] Finally, the calculated momentum information is converted into momentum embedding. Momentum embedding is a low-dimensional representation that captures motion information in a sequence of video frames. The generation of momentum embedding can be achieved by using machine learning algorithms such as neural networks or dimensionality reduction techniques such as principal component analysis. The generated momentum embedding will be used as one of the inputs for subsequent processing.

[0126] S42, performing spectrum embedding processing according to the spectrum mask.

[0127] In specific implementation, step S42, performing spectrum embedding processing according to the spectrum mask may further include the following steps:

[0128] S421, extracting a spectrum mask.

[0129] In the previous steps, different spectral masks have been generated. Spectral masks are used to separate different sound sources in mixed audio. In this step, the spectral mask needs to be extracted from the feature fusion result. The extraction of the spectral mask can be achieved by thresholding or clustering analysis of the feature fusion result.

[0130] S422, performing short-time Fourier transform on the audio.

[0131] After extracting the spectrum mask, the mixed audio needs to be subjected to a short-time Fourier transform (STFT). STFT is a method that converts a time domain signal into a frequency domain signal, and is used to analyze the changes in the audio signal over time and frequency. Through STFT, a spectrogram of the audio signal can be obtained, which reflects the energy distribution of the audio signal at different frequencies.

[0132] S423, extracting spectrum features.

[0133] After obtaining the spectrum graph, it is necessary to extract spectrum features. Spectral features refer to the statistical characteristics of audio signals in the frequency domain, such as spectrum energy, spectrum centroid, spectrum flatness, etc. These features can be used to describe the characteristics of audio signals in the frequency domain and serve as input for subsequent processing.

[0134] S424, perform spectrum embedding according to the spectrum mask.

[0135] After extracting the spectral features, spectral embedding needs to be performed based on the spectral mask. Spectral embedding is a method of converting spectral features into low-dimensional representations to capture key information in audio signals. The purpose of spectral embedding based on spectral mask is to separate different sound sources in mixed audio and generate corresponding spectral embeddings.

[0136] The generation of spectrum embedding can be achieved through the following steps:

[0137] Mask application,Apply the spectral mask to the spectrogram to extract the spectral features of different sound sources.,This step can be achieved by element-wise multiplication of the spectral mask and the,spectrogram.

[0138] Feature extraction: extract features from the masked spectrum to obtain the spectrum features of different sound sources. These features can be statistical properties such as spectrum energy and spectrum centroid.

[0139] Embedding generation, which converts the extracted spectral features into spectral embeddings. The generation of spectral embeddings can be achieved by using machine learning algorithms (such as neural networks) or dimensionality reduction techniques (such as principal component analysis).

[0140] In step S5, the momentum embedding and the spectrum embedding are respectively input into the momentum-aware Transformer module to generate sequence information.

[0141] Momentum embedding is the motion information extracted from consecutive video frames, while spectral embedding is the frequency domain features extracted from mixed audio files. These two embedding methods capture the key information in video and audio respectively.

[0142] Momentum embedding captures the motion changes between video frames, which helps to identify the dynamic behavior of the respondent. Spectral embedding captures the frequency components in the audio signal, which helps to identify the voice characteristics of the respondent.

[0143] In this embodiment, step S5, respectively inputting momentum embedding and spectrum embedding into the momentum-aware Transformer module, and generating sequence information may further include the following steps:

[0144] S51, the momentum embedding and spectrum embedding are input into the momentum-aware Transformer module.

[0145] S511, extract momentum embedding.

[0146] The process of extracting momentum embedding involves calculating the momentum information of consecutive video frames and generating momentum embeddings. The momentum information can be achieved by calculating the pixel difference or optical flow between adjacent frames. Then, this momentum information is encoded into an embedding vector for input into the momentum-aware Transformer module.

[0147] The functions of extracting momentum embedding are: capturing motion. Momentum embedding can capture the motion changes between video frames, which is crucial for identifying the dynamic behavior of the respondent; enhancing video information. By encoding momentum information, the motion features in the video frames can be effectively utilized by the model.

[0148] S512, extracting spectrum embedding.

[0149] The extraction process of spectral embedding involves performing a short-time Fourier transform (STFT) on the mixed audio file and then extracting features from the transformed audio. These features can include spectral amplitude, spectral phase, Mel-frequency cepstral coefficients (MFCC), etc. These features are encoded into an embedding vector for input into the momentum-aware Transformer module.

[0150] The functions of extracting spectral embedding are: capturing audio features. Spectral embedding can capture the frequency components in the audio signal, which is crucial for identifying the voice characteristics of the respondent; enhancing audio information. By encoding spectral features, the key information in the audio signal can be effectively used by the model.

[0151] S52, apply the momentum-aware Transformer module.

[0152] The momentum-aware Transformer module is a deep learning model based on the self-attention mechanism, which can efficiently model the sequence of input embeddings. The core of this module is the self-attention mechanism, which allows the model to take into account the information of all other positions in the sequence when processing the current input.

[0153] Self-attention mechanism: By performing weighted summation of the embedding vectors, a new representation is generated, allowing the model to capture long-distance dependencies in the sequence.

[0154] Momentum perception: Through a specially designed attention mechanism, the model can better handle embeddings containing motion information and improve the accuracy of sound source localization.

[0155] S521, using self-attention mechanism.

[0156] The core of the momentum-aware Transformer module is the self-attention mechanism. The self-attention mechanism allows the model to take into account information from all other positions in the sequence when processing the current input. This is achieved by computing dot product attention scores between the input embeddings, and then using these scores to perform a weighted sum of the input embeddings to generate a new representation.

[0157] The functions of step S521 are: capturing long-distance dependencies. The self-attention mechanism can capture long-distance dependencies between input embeddings, which is crucial for understanding the global context in the sequence; enhancing the ability to express information. Through weighted summation, the model can generate representations that contain global information, thereby enhancing the ability to express information.

[0158] S522, Design momentum perception.

[0159] The momentum-aware Transformer module specifically designs an attention mechanism for momentum embeddings. This design enables the model to more effectively capture and utilize this information when processing embeddings containing motion information. For example, this can be done by increasing the attention weight on momentum embeddings or designing a dedicated attention head to process momentum embeddings.

[0160] The functions of step S522 are: improving accuracy. Through a specially designed attention mechanism, the model can better handle embeddings containing motion information and improve the accuracy of sound source localization; adapting to dynamic environments, so that the model can adapt to dynamic teaching environments and better deal with various dynamic behaviors that may occur in the respondent.

[0161] S53, generating sequence information.

[0162] After the momentum-aware Transformer module processes the input embedding, it outputs a series of new representations that contain the sequence information of the input embedding. This sequence information contains not only the information in the original embedding, but also the sequence dependencies captured by the self-attention mechanism.

[0163] The sequence information provides the temporal relationship between the input embeddings, which is crucial for subsequent spectral mask generation and audio separation.

[0164] Enhanced information: Through the self-attention mechanism, the model can learn the potential connections between input embeddings and enhance the expressiveness of information.

[0165] The sequence information output by the momentum-aware Transformer module is composed of a series of new representations. These representations contain the original information in the input embeddings, as well as the sequence dependencies captured by the self-attention mechanism. These new representations can be regarded as enhanced versions of the input embeddings, which contain richer and more accurate information.

[0166] The sequence information provides the temporal relationship between the input embeddings, which is crucial for subsequent spectral mask generation and audio separation.

[0167] Through the self-attention mechanism, the model can learn the potential connections between input embeddings and enhance the quality of information.

[0168] The generated sequence information will be used in the subsequent spectral mask generation and audio separation steps. In the spectral mask generation stage, the sequence information will be used to guide the model to generate masks that can accurately distinguish different sound sources. In the audio separation stage, these masks will be used to extract the voice of the respondent from the mixed audio file.

[0169] The sequence information provides guidance for the generation of spectral masks, allowing the model to generate more accurate and robust masks. By leveraging the generated masks, the model is able to accurately extract the voice of the respondent from the mixed audio file.

[0170] Step S6, performing residual fusion on the sequence information and the feature fusion result to generate a spectrum mask.

[0171] In this embodiment, step S6, performing residual fusion on the sequence information and the feature fusion result to generate a spectrum mask may specifically include the following steps:

[0172] S61, design residual fusion strategy.

[0173] The design of the residual fusion strategy needs to consider the characteristics of the input data and the fusion target. In this embodiment, addition fusion or multiplication fusion can be used.

[0174] Additive fusion: directly add the sequence information and feature fusion results, which is suitable for situations where the information of the two is highly complementary.

[0175] Multiplication fusion: multiply the sequence information and feature fusion results element by element, which is suitable for situations where there is interaction between the two information.

[0176] S62, performing a residual fusion operation.

[0177] According to the designed fusion strategy, the sequence information and feature fusion results are fused. During the fusion process, it is necessary to ensure data alignment and synchronization to avoid information loss or misalignment.

[0178] The fused results can be preliminarily checked to ensure the integrity and consistency of the data.

[0179] S63: Generate a spectrum mask.

[0180] Spectral mask is a tool for audio separation and processing, which retains or suppresses specific frequency components by applying a mask on the spectrum graph. In this embodiment, the result of residual fusion is used to generate a spectral mask.

[0181] Mask generation method: Based on the results of residual fusion, a binary or continuous valued spectral mask is generated through methods such as threshold segmentation, clustering analysis or deep learning model.

[0182] The mask can be optimized according to actual needs: the generated spectrum mask can be post-processed, such as smoothing and denoising, to improve the quality and stability of the mask. According to the actual test results, the mask generation method can be iteratively optimized to obtain better audio separation and positioning effects.

[0183] Step S7, performing audio separation according to the spectrum mask, and generating a visual sound source localization map through CAM.

[0184] In this embodiment, step S7, performing audio separation according to the spectrum mask and generating a visual sound source localization map through CAM, may specifically include the following steps:

[0185] S71, apply spectrum mask.

[0186] Load the spectral mask generated in the previous step. The spectral mask is a two-dimensional array, each element of which has a value between 0 and 1, indicating the degree of signal preservation at the corresponding frequency and time period.

[0187] Assume the spectral mask is already saved as a numpy array.

[0188] import numpy as np;

[0189] spectrum_mask = np.load(spectrum_mask.npy).

[0190] A spectral mask is applied to the short-time Fourier transform (STFT) result of the mixed audio to separate the spectrum of the target sound source.

[0191] Assume that the STFT result of the mixed audio has been calculated and saved in the variable stft_result;

[0192] stft_result_separated = stft_resultspectrum_mask.

[0193] S72, performing inverse transformation on the audio signal.

[0194] The separated spectrum is converted back to the time domain signal through inverse short-time Fourier transform (ISTFT) to obtain the audio signal of the target sound source.

[0195] For example use scipy.signal.istft to do the inverse transform.

[0196] import scipy.signal as signal;

[0197] audio_signal_separated = signal.istft(stft_result_separated).

[0198] During the ISTFT process, boundary effects may be generated, which can be processed to improve the quality of the audio signal. For example, the overlap-add method (Overlap-Add) or the overlap-save method (Overlap-Save) can be used to process the boundary effects.

[0199] This embodiment takes the overlap-add method as an example, assuming that the length of each frame is frame_length and the overlap length is overlap_length, and overlap-adds the divided audio signals. The specific implementation can be adjusted according to actual conditions.

[0200] S73, Class Activation Mapping (CAM) generates a visual sound source localization map.

[0201] S731, extracting features from the separated audio signal, and using the extracted features to generate a CAM.

[0202] You can use audio processing libraries such as librosa to extract features: import librosa.

[0203] Take the extraction of MFCC features as an example:

[0204] mfcc_features = librosa.feature.mfcc(y=audio_signal_separated, sr=sr,n_mfcc=13).

[0205] S732, build a classification model.

[0206] Build a classification model that can classify audio signals based on the extracted features and output category scores.

[0207] Let’s take the convolutional neural network (CNN) as an example:

[0208] from keras.models import Sequential;

[0209] from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense;

[0210] model = Sequential();

[0211] model.add(Conv2D(32, (3, 3), activation=relu, input_shape=(mfcc_features.shape[1], mfcc_features.shape[0],1)));

[0212] model.add(MaxPooling2D((2, 2)));

[0213] model.add(Flatten());

[0214] model.add(Dense(10, activation=softmax)) Assume there are 10 categories,

[0215] Compile the model model.compile(optimizer=adam, loss=categorical_crossentropy, metrics=[accuracy]).

[0216] S733, calculate the category activation map (CAM).

[0217] Based on the output of the classification model and the feature map, a class activation map (CAM) is calculated to visualize the location of the target sound source in the audio signal.

[0218] Assume that the output of the last convolutional layer of the classification model is feature_maps, the output of the global_average_pooling layer is pool_output, and the class scores (logits) are logits.

[0219] def generate_cam(feature_maps, weights, input_shape);

[0220] cam = np.zeros(input_shape[:2], dtype=np.float32);

[0221] for i, w inenumerate(weights);

[0222] cam += wfeature_maps[:, :, i];

[0223] cam = np.maximum(cam);

[0224] cam = cam / cam.max() normalized;

[0225] cam = np.uint8(255cam) # Convert to 8-bit image;

[0226] return cam.

[0227] Get feature maps and category scores:

[0228] feature_maps = model.predict(np.expand_dims(mfcc_features, axis=-1))[0] Assume there is only one sample.

[0229] logits = model.output, get the logits layer.

[0230] Assuming we focus on the first category (index 0):

[0231] category_index = 0;

[0232] weights = model.layers[-1].get_weights()[0][category_index] gets the weight of the corresponding category of the softmax layer.

[0233] Calculate CAM:

[0234] cam =generate_cam(feature_maps, weights, mfcc_features.shape[:2]).

[0235] Visual CAM:

[0236] import matplotlib.pyplot as plt;

[0237] plt.imshow(cam, cmap=jet);

[0238] plt.title(Class ActivationMapping);

[0239] plt.show().

[0240] S734, mapping to video frame.

[0241] The CAM results are mapped into a video frame sequence to generate a visual sound source localization map.

[0242] Assume that the video frame sequence has been loaded as a frames list, and the size of each frame is (frame_height, frame_width).

[0243] frame_height,frame_width =frames[0].shape[:2].

[0244] Scale the CAM result to the size of the video frame:

[0245] cam_resized = cv2.resize(cam, (frame_width,frame_height)).

[0246] Overlay the CAM results onto the video frame to generate a visual sound source localization map:

[0247] for i, frame in enumerate(frames);

[0248] vis_frame = frame.copy();

[0249] vis_frame[cam_resized>128]= [0, 0, 255] displays high brightness areas in red.

[0250] # Or use other methods to overlay CAM results, such as transparency overlay.

[0251] # cv2.addWeighted(frame, alpha, cam_resized, beta, gamma, dst=vis_frame).

[0252] # Display or save the visualization results.

[0253] cv2.imshow(Visualized Frame, vis_frame);

[0254] cv2.waitKey(1) displays frame by frame.

[0255] # Or save as an image file.

[0256] # cv2.imwrite(fvisualized_frame_{i}.png, vis_frame).

[0257] Release window resources:

[0258] cv2.destroyAllWindows().

[0259] For example, the above embodiment is implemented in the classroom teaching of securities investment funds. According to the classroom teaching scenario of the knowledge quiz competition, students can be divided into groups first, and students in the same group sit together, with 5-7 people in each group. The quiz is divided into group quiz before class and individual quiz in class. However, no matter which form of quiz, its final assessment is closely related to the collective performance of the group. Those who quiz and answer correctly more times will have a higher overall cumulative score in the end.

[0260] In the quick-answer session, the present embodiment is applied to enable the teacher to quickly and accurately identify which student answers the question the fastest.

[0261] The first part is the audio-appearance model, which uses purely appearance-based features to separate sound sources. First, a frame is randomly extracted from the video sequence and encoded into a dimension of The input audio mixture is converted to The spectrum image is then encoded into a size of The obtained feature volume is multiplied by the visual feature vector by channel and converted to The output in audio-appearance mode is formed by multiplying the mixed spectrogram with the binary mask.

[0262] The above results are further passed to the momentum-aware module, which uses motion cues to refine source separation. The corresponding features are extracted by the motion network, which encodes the video sequence into a motion representation of size T'×CM, where T' corresponds to the temporal dimension of the sequence.

[0263] No assumptions are made about the type of motion (e.g., human motion). The subsequent Sound Spectrogram Refinement Network (SSR) network combines the motion features with the output spectrogram of the previous stage. The core part of the refinement network is the Audio-Video Momentum Transformer (AV-Motion Transformer, AME) module, which uses a multimodal Transformer architecture to associate motion representations with spectrogram features. The final output is formed by multiplying the original mixed spectrogram with the binary mask obtained from the refinement network. In this way, teachers can quickly lock in the student who answers the question the fastest during the quiz.

[0264] The implementation of this embodiment has the following beneficial effects: by combining the key frame features of the video frame and the short-time Fourier transform features of the audio, full use is made of visual and auditory information, a more comprehensive scene understanding is achieved, and the accuracy of sound source localization is improved; by calculating the momentum information of continuous video frames and embedding the audio spectrum, key information in dynamic changes can be captured, the problem of sound source movement in complex environments can be effectively dealt with, and the real-time response capability of the positioning system can be improved; the sequence information generated by the momentum-aware Transformer is residually fused with the multimodal fusion result, the feature representation is further optimized, and the final generated spectrum mask is more accurate, which helps to separate the target sound source; the visual sound source localization map generated by CAM (class activation mapping) not only provides intuitive feedback for teaching, but also facilitates students to understand the principle and process of sound source localization, enhances the interactivity and fun of learning, and helps to improve teaching efficiency and quality.

[0265] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0266] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0267] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0268] Embodiment 2

[0269] Further references Figure 2 , as a response to the above Figure 1 The present invention provides an embodiment of a teaching interactive answering positioning device, and the device embodiment is Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0270] like Figure 2 As shown, the teaching interactive answering positioning device 80 described in this embodiment includes: an acquisition module 81, a feature extraction module 82, a multimodal fusion module 83, a processing module 84, a generation module 85, a residual fusion module 86 and a positioning module 87. Among them:

[0271] An acquisition module 81 is used to acquire input data, wherein the input data includes a set of video frame sequences and a corresponding mixed audio file;

[0272] A feature extraction module 82 is used to identify key frames in the set of the video frame sequence, perform feature extraction on the key frames to obtain first modal information, perform short-time Fourier transform on the mixed audio, and then perform feature extraction on the audio to obtain second modal information;

[0273] A multimodal fusion module 83, configured to perform multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result and generate different spectrum masks;

[0274] A processing module 84, configured to calculate momentum information of consecutive video frames and generate momentum embedding, and perform spectrum embedding processing according to the spectrum mask;

[0275] A generating module 85, for respectively inputting momentum embedding and spectrum embedding into a momentum-aware Transformer module to generate sequence information;

[0276] A residual fusion module 86, configured to perform residual fusion on the sequence information and the feature fusion result to generate a spectrum mask;

[0277] The localization module 87 is used to perform audio separation according to the spectrum mask and generate a visual sound source localization map through CAM.

[0278] The implementation of this embodiment has the following beneficial effects: by combining the key frame features of the video frame and the short-time Fourier transform features of the audio, full use is made of visual and auditory information, a more comprehensive scene understanding is achieved, and the accuracy of sound source localization is improved; by calculating the momentum information of continuous video frames and embedding the audio spectrum, key information in dynamic changes can be captured, the problem of sound source movement in complex environments can be effectively dealt with, and the real-time response capability of the positioning system can be improved; the sequence information generated by the momentum-aware Transformer is residually fused with the multimodal fusion result, the feature representation is further optimized, and the final generated spectrum mask is more accurate, which helps to separate the target sound source; the visual sound source localization map generated by CAM (class activation mapping) not only provides intuitive feedback for teaching, but also facilitates students to understand the principle and process of sound source localization, enhances the interactivity and fun of learning, and helps to improve teaching efficiency and quality.

[0279] Embodiment 3

[0280] The teaching interactive answering positioning device described in this embodiment may also include an appearance module, a sound feature extraction network, a sound source separation module, a motion network, a sound network, an audio-video momentum Transformer (Audio-Motion Transformer, AMT) module, an audio-video embedding module, an encoder and decoder module, a residual fusion module, etc.

[0281] The appearance module receives a random single frame of the input video and applies the dilated Res18-2D to obtain a compact semantic representation. More specifically, given an input RGB image, the appearance network produces a Representation , as the output of the last spatial average pooling layer.

[0282] The sound feature extraction network first converts the input audio waveform into a spectrogram representation using short-time Fourier transform (STFT). The sound spectrogram network will Encoded into a set of feature maps. The network is implemented using the MobileNetV2 (MV2) architecture and encodes a The input spectrum is converted to a size of Feature map .

[0283] The number of feature maps generated Equal to the appearance feature vector dimension The audio feature separation process is shown in Equation 3-2. In the separation process, the mixed audio Feature extraction is performed through a multi-layer neural network, and feature data is generated. After that, it is calculated with the visual features to obtain the spectrum mask of the separated target.

[0284] The sound source separation module uses feature maps and to produce an estimate of the component audio corresponding to the input video. More specifically,

[0285]

[0286] in, and denote the channel-wise and element-wise products, respectively. represent Operation. If ,but , otherwise 0. It is Source (No. The appearance network output of the input video is obtained by Mixed spectrum with the original Perform element-wise multiplication to obtain the output spectrum .

[0287] Next, the Motion Network The input video frame Map the input video sequence to a vector space. (Size is ) into a 3D version of Res18 and generate a size of Representation ,in , , Through additional 3D convolution, a single-channel feature map is obtained , the size is The final embedding vector is obtained by applying spatial average pooling , the size is .

[0288] Sound Network Mapping audio waveforms to motion networks Shared embedding space. You can use the Res18-1D architecture, which consists of a series of convolutions with stride 1D, applied until the size of the output representation matches the motion network output ,Right now .

[0289] Leverage temporal alignment of natural audio and motion to train an audio-video embedding module. Given a video clip, aligned audio and unaligned audio }, and their corresponding embeddings , and , define a ternary loss function as follows:

[0290]

[0291] in, Measure the similarity between motion and audio embeddings. In all experiments, we use The embedding network is optimized with respect to a loss function over a large set of input triplets. This objective function encourages the motion network to focus primarily on motion features in the input video that are related to sound. This loss function is minimized over a large set of videos to optimize the embedding network.

[0292] The motion stage uses motion cues to separate sound sources. This stage consists of four components: motion network, SSR network, audio and video momentum Transformer, and residual fusion module.

[0293] The Sound Spectrogram Refinement (SSR) network is an encoder-decoder architecture consisting of 7 down-convolutional layers and 7 up-convolutional layers, followed by a BatchNorm layer and a Leaky ReLU layer. ) receives the spectrogram from the audio-appearance stage , and generate feature representation (The superscript motion refers to the audio-motion stage.) The encoder is followed by the Audio-Video Momentum Transformer (AMT), which transforms With sports characteristics Fusion This is followed by a spatial pooling, which is generated by the motion network of AME in Section 4.2). The transformed output of AMT Passed to the up-convolution decoder ( ) to produce a size of The residual spectrum representation of . A filter method is used to move the identified residual sound components from the audio-appearance output to the final corresponding output. The process of the SSR network is defined as Equation 3-3:

[0294]

[0295] The Audio-Motion Transformer (AMT) module is used to perform sound source separation using motion cues learned (the motion of the sound learned through AME). The overall architecture is visualized using CAM. First, the input embeddings are positionally encoded to preserve the order of the sequence. The following encoder and decoder modules consist of stacked multi-head attention, point-to-point, and fully connected layers.

[0296] The encoder will Source Convert to query ,key and value input. This has the same , and The multi-head attention of is often called self-multi-head attention. Similarly, at the beginning of the decoder, AMT has Sound characteristics Self-multi-head attention ( , and ). After the self-attention process, the decoder incorporates the motion into the sound component through the audio-video momentum multi-head attention, which Sound embedding from a source Treat as a query , will Motion embedding of the source Treat as key Sum ,in , N is the number of sources.

[0297] The residual fusion module (RFM) Output (Equation 3-3) is parsed as a residual spectrum. That is, it represents the part of the appearance output (Equation 1) that needs to be redistributed. For example, Defines the source But currently assigned to source Reposition these components between the spectrograms as follows:

[0298]

[0299] in, It is The output spectrum of the audio-appearance stage for each source, From the sound To Sound The residual spectrum. and Represents the sum and difference of element dimensions respectively. The feature map obtained pass And threshold operation, the final spectrum output It is formed by multiplying the original mixed spectrogram with the result obtained. An inverse STFT is applied to produce the final separated audio waveform.

[0300] Formulate the overall learning goal as a binary mask , these masks are used to obtain the final output spectrogram (Equations 3-2 and 3-4). The real mask The formation is as follows,

[0301]

[0302] in, Represents a sound spectrum The time-frequency coordinates in . The binary mask estimated by minimizing and the true binary mask The model is trained using the binary cross entropy (BCE) loss between:

[0303]

[0304] in, and They represent the binary masks predicted in the appearance and momentum stages respectively. Fusion of the binary mask with the original audio can yield the separated audio spectrum.

[0305] The implementation of this embodiment has the following beneficial effects: by combining the key frame features of the video frame and the short-time Fourier transform features of the audio, full use is made of visual and auditory information, a more comprehensive scene understanding is achieved, and the accuracy of sound source localization is improved; by calculating the momentum information of continuous video frames and embedding the audio spectrum, key information in dynamic changes can be captured, the problem of sound source movement in complex environments can be effectively dealt with, and the real-time response capability of the positioning system can be improved; the sequence information generated by the momentum-aware Transformer is residually fused with the multimodal fusion result, the feature representation is further optimized, and the final generated spectrum mask is more accurate, which helps to separate the target sound source; the visual sound source localization map generated by CAM (class activation mapping) not only provides intuitive feedback for teaching, but also facilitates students to understand the principle and process of sound source localization, enhances the interactivity and fun of learning, and helps to improve teaching efficiency and quality.

[0306] Embodiment 4

[0307] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0308] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 having components of a memory 91, a processor 92, and a network interface 93, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0309] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0310] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 may be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 9. Of course, the memory 91 may also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of the teaching interactive answering positioning method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0311] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run the computer-readable instructions stored in the memory 91 or process data, such as running the computer-readable instructions of the teaching interactive answering positioning method.

[0312] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0313] The implementation of this embodiment has the following beneficial effects: by combining the key frame features of the video frame and the short-time Fourier transform features of the audio, full use is made of visual and auditory information, a more comprehensive scene understanding is achieved, and the accuracy of sound source localization is improved; by calculating the momentum information of continuous video frames and embedding the audio spectrum, key information in dynamic changes can be captured, the problem of sound source movement in complex environments can be effectively dealt with, and the real-time response capability of the positioning system can be improved; the sequence information generated by the momentum-aware Transformer is residually fused with the multimodal fusion result, the feature representation is further optimized, and the final generated spectrum mask is more accurate, which helps to separate the target sound source; the visual sound source localization map generated by CAM (class activation mapping) not only provides intuitive feedback for teaching, but also facilitates students to understand the principle and process of sound source localization, enhances the interactivity and fun of learning, and helps to improve teaching efficiency and quality.

[0314] Embodiment 5

[0315] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the teaching interactive answering positioning method as described above.

[0316] The implementation of this embodiment has the following beneficial effects: by combining the key frame features of the video frame and the short-time Fourier transform features of the audio, full use is made of visual and auditory information, a more comprehensive scene understanding is achieved, and the accuracy of sound source localization is improved; by calculating the momentum information of continuous video frames and embedding the audio spectrum, key information in dynamic changes can be captured, the problem of sound source movement in complex environments can be effectively dealt with, and the real-time response capability of the positioning system can be improved; the sequence information generated by the momentum-aware Transformer is residually fused with the multimodal fusion result, the feature representation is further optimized, and the final generated spectrum mask is more accurate, which helps to separate the target sound source; the visual sound source localization map generated by CAM (class activation mapping) not only provides intuitive feedback for teaching, but also facilitates students to understand the principle and process of sound source localization, enhances the interactivity and fun of learning, and helps to improve teaching efficiency and quality.

[0317] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the various embodiment methods of the present invention.

[0318] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to perform equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A teaching interactive answering positioning method, characterized in that: The steps include: Acquire input data, the input data comprising a set of video frame sequences and a corresponding mixed audio file; Identifying key frames in the set of the video frame sequence, performing feature extraction on the key frames to obtain first modal information, performing short-time Fourier transform on the mixed audio, and then performing feature extraction on the audio to obtain second modal information; Performing multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result, and generating different spectrum masks; Calculating momentum information of consecutive video frames and generating momentum embedding, and performing spectrum embedding processing according to the spectrum mask; The momentum embedding and spectrum embedding are respectively input into the momentum-aware Transformer module to generate sequence information; Performing residual fusion on the sequence information and the feature fusion result to generate a spectrum mask; Audio separation is performed according to the spectrum mask, and a visual sound source localization map is generated through CAM.

2. The teaching interactive answering positioning method according to claim 1 is characterized in that: The step of identifying key frames in the set of the video frame sequence, performing feature extraction on the key frames to obtain the first modal information, performing short-time Fourier transform on the mixed audio, and then performing feature extraction on the audio to obtain the second modal information specifically includes: identifying the key frame in the set of video frame sequences; Performing feature extraction on the key frame to obtain first modal information; Perform short-time Fourier transform (STFT) on the mixed audio, and then extract features from the audio to obtain the second modal information.

3. The teaching interactive answering positioning method according to claim 1 is characterized in that: The step of performing multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result and generate different spectrum masks specifically includes: Preprocessing the first modal information and the second modal information; Align video frame features and audio features; Select feature fusion strategy; Fuse the video frame features and audio features and generate feature fusion results; Generate a spectral mask.

4. The teaching interactive answering positioning method according to claim 1 is characterized in that: The step of calculating the momentum information of the continuous video frames and generating the momentum embedding, and performing spectrum embedding processing according to the spectrum mask specifically comprises: Calculate momentum information of consecutive video frames; Perform spectrum embedding processing according to the spectrum mask.

5. The teaching interactive answering positioning method according to claim 1, characterized in that: The step of respectively embedding momentum and spectrum into the momentum-aware Transformer module to generate sequence information specifically includes: Input momentum embedding and spectral embedding into the momentum-aware Transformer module; Apply momentum-aware Transformer module; Generate sequence information.

6. The teaching interactive answering positioning method according to claim 1, characterized in that: The step of performing residual fusion on the sequence information and the feature fusion result to generate a spectrum mask specifically includes: Design residual fusion strategy; Perform residual fusion operation; Generate a spectral mask.

7. The teaching interactive answering positioning method according to any one of claims 1 to 6, characterized in that: The step of performing audio separation according to the spectrum mask and generating a visual sound source localization map through CAM specifically includes: Applying a spectral mask; Perform inverse transformation on the audio signal; Class Activation Mapping (CAM) generates visual sound source localization maps.

8. A teaching interactive answering positioning device, characterized in that: include: An acquisition module, used to acquire input data, wherein the input data includes a set of video frame sequences and a corresponding mixed audio file; A feature extraction module, used to identify key frames in the set of the video frame sequence, perform feature extraction on the key frames to obtain first modal information, perform short-time Fourier transform on the mixed audio, and then perform feature extraction on the audio to obtain second modal information; A multimodal fusion module, configured to perform multimodal fusion on the first modal information and the second modal information to obtain a feature fusion result and generate different spectrum masks; A processing module, used for calculating momentum information of consecutive video frames and generating momentum embedding, and performing spectrum embedding processing according to the spectrum mask; A generation module, which is used to input momentum embedding and spectrum embedding into the momentum-aware Transformer module to generate sequence information; A residual fusion module, used for performing residual fusion on the sequence information and the feature fusion result to generate a spectrum mask; The localization module is used to perform audio separation according to the spectrum mask and generate a visual sound source localization map through CAM.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the teaching interactive answering positioning method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the teaching interactive answering positioning method as described in any one of claims 1 to 7 are implemented.

Images