A method and system for analyzing teachers' teaching speech behavior in multimodal classrooms

By performing segmented analysis of classroom voice signals and filtering weight calculation, adaptive noise reduction processing is realized, which solves the problem of noise interference in classroom teaching environment and improves the accuracy of teacher speech behavior analysis.

CN118629417BActive Publication Date: 2024-11-22HUAZHONG NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411104190.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-22
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively reduce noise in classroom teaching environments, resulting in a decrease in the accuracy of teacher speech data analysis and affecting the effectiveness of speech behavior analysis.

Method used

Multimodal classroom teachers' teaching speech behavior analysis method is adopted, and by obtaining classroom speech signals and component speech signals, performing segmented analysis, calculating the change contribution degree and filtering weight, and adaptive noise reduction processing is performed in combination with noise intensity.

Benefits of technology

It improves the noise reduction effect of classroom voice signals and enhances the accuracy and effectiveness of teacher teaching speech behavior analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118629417B_ABST
    Figure CN118629417B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of speech analysis and processing technology, and specifically to a multimodal classroom teacher teaching speech behavior analysis method and system. The present invention first obtains a classroom speech signal and all component speech signals of the classroom speech signal, and performs segment numbering; obtains the change contribution of each component speech segment, obtains the first filter weight and the second filter weight of each component speech segment, and then combines the noise intensity of the classroom speech signal to reduce the noise of all component speech signals; and performs speech behavior analysis based on all component speech signals after noise reduction. The present invention analyzes the classroom speech signal and all its component speech signals in segments, evaluates the influence of noise interference on each segment to determine the first filter weight and the second filter weight, and then performs adaptive filtering on each segment to improve the noise reduction effect of the classroom speech signal, thereby improving the speech behavior analysis effect of the teacher's classroom teaching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of speech analysis and processing, and in particular to a method and system for analyzing the teaching speech behavior of a multimodal classroom teacher. Background Art

[0002] The voice data of teachers in the classroom teaching process contains rich information, including speaking speed, intonation, emotional expression and other information that can be used to analyze the teacher's speech behavior. By analyzing and processing the voice data, the teacher's language characteristics and emotional expression can be identified, such as whether the teacher has emotional fluctuations during the lecture, whether the language expression is vivid and interesting to attract students' interest in learning, etc. This helps to have a deeper understanding of the teacher's speech behavior and provide a scientific basis for teaching evaluation and improvement.

[0003] However, in the actual teaching environment, there are various background interferences, such as air conditioning noise, keyboard tapping, student voices, etc. These interferences will reduce the accuracy of the teacher's voice data collection and analysis. Therefore, the signal decomposition algorithm is usually used to smooth each component signal to eliminate the influence of noise interference. However, the interference factors in the decomposition results are scattered in different component results. Conventional noise removal methods cannot distinguish the actual voice information from the interference information, thus affecting the effect of speech behavior analysis. Summary of the invention

[0004] In order to solve the technical problem that the existing technology has poor noise reduction effect on classroom voice data, resulting in poor analysis effect of classroom teacher teaching speech behavior, the purpose of the present invention is to provide a multimodal classroom teacher teaching speech behavior analysis method and system, and the technical solution adopted is as follows:

[0005] The present invention proposes a method for analyzing the teaching speech behavior of a multimodal classroom teacher, the method comprising:

[0006] Acquire a classroom voice signal and all component voice signals of the classroom voice signal; segment and number the classroom voice signal and all the component voice signals to obtain all classroom voice segments and component voice segments, wherein the classroom voice segments and component voice segments with the same sequence number have the same length;

[0007] According to the changes in the distribution characteristics of the component speech segments and the classroom speech segments in each of the component speech signals, the change contribution of each of the component speech segments is obtained; according to the difference in the change contribution between different component speech segments under the same sequence number, combined with the contribution rate of each component speech segment in all the component speech segments under the same sequence number, the first filtering weight of each component speech segment is obtained; according to the time-frequency distribution characteristics of each component speech segment of each component speech signal, the second filtering weight of each component speech segment is obtained; according to the first filtering weight and the second filtering weight, combined with the noise intensity of the classroom speech signal, all the component speech signals are denoised;

[0008] Conduct teaching speech behavior analysis based on the noise reduction results.

[0009] Furthermore, the method for obtaining the change contribution includes:

[0010] According to the frequency resolution of each speech segment and the distribution disorder of the signal points, the distribution characteristic parameters of each speech segment are obtained;

[0011] In each of the component speech signals, taking the difference of the distribution characteristic parameters between each of the component speech segments and the next adjacent component speech segment as a first parameter difference;

[0012] In the classroom speech signal, the difference of the distribution characteristic parameters between each classroom speech segment and the next adjacent classroom speech segment is used as the second parameter difference;

[0013] Between speech segments corresponding to the same sequence number, the ratio of the first parameter difference to the second parameter difference is used as the change contribution of the corresponding component speech segment.

[0014] Furthermore, the method for acquiring the distribution characteristic parameters includes:

[0015] Obtain the information entropy of each speech segment;

[0016] Obtaining extreme value points in each speech segment; in each speech segment, calculating the time intervals between adjacent extreme value points, and taking the average of all the time intervals as the frequency resolution of the speech segment;

[0017] According to the information entropy and the frequency resolution of each speech segment, the distribution characteristic parameters of each speech segment are obtained, and the information entropy and the frequency resolution are both positively correlated with the distribution characteristic parameters.

[0018] Furthermore, the method for obtaining the first filtering weight includes:

[0019] Taking any of the component speech segments as a target segment, obtaining reconstructed speech segments of all the component speech segments except the target segment under the same sequence number, and taking the difference in the distribution feature parameters between the reconstructed speech segment and the classroom speech segment under the same sequence number as the importance parameter of the target segment;

[0020] Under the same sequence number, obtaining an average value of the difference in the change contribution between the target segment and all the other component speech segments, and taking the average value as the abnormal parameter of the target segment;

[0021] According to the importance parameter and the abnormal parameter of each of the component speech segments, a first filtering weight of each of the component speech segments is obtained, and both the importance parameter and the abnormal parameter are positively correlated with the first filtering weight.

[0022] Furthermore, the method for acquiring the second filtering weight includes:

[0023] Performing Fourier transformation on each of the component speech segments to obtain a frequency domain component speech segment of each of the component speech segments; obtaining a segment similarity parameter of each of the component speech signals according to a distribution probability difference between signal points with the same sequence number in different frequency domain component speech segments;

[0024] The amplitude variance of the signal point in each of the component speech segments is used as the time domain amplitude variance, and the amplitude of each signal point in the component speech segment is weighted and averaged according to the time domain amplitude variance to obtain the time domain feature parameter of each of the component speech segments; the amplitude variance of the signal point in each of the frequency domain component speech segments is used as the frequency domain amplitude variance, and the amplitude of each signal point in the frequency domain component speech segment is weighted and averaged according to the frequency domain amplitude variance to obtain the frequency domain feature parameter of each of the component speech segments;

[0025] Taking the difference between the time domain feature parameter and the frequency domain feature parameter of each of the component speech segments as the time-frequency distribution consistency parameter of each of the component speech segments;

[0026] According to the segmented similarity parameter and the time-frequency distribution consistency parameter, a second filtering weight of each component speech signal is obtained, and the segmented similarity parameter and the time-frequency distribution consistency parameter are both positively correlated with the second filtering weight.

[0027] Furthermore, the calculation formula of the segment similarity parameter includes:

[0028] ;in, For the The piecewise similarity parameters of the component speech signals; For the The combined serial number of the frequency domain component speech segments corresponding to any two component speech segments in the component speech signal; For the The total number of combinations of frequency domain component speech segments corresponding to any two component speech segments of a component speech signal; is the serial number of the signal point in each frequency domain component speech segment; is the total number of signal points in the speech segment for each frequency domain component; For the The first frequency domain component speech segment The amplitude of the signal point; For the The first frequency domain component speech segment The amplitude of the signal point; is a logarithmic function.

[0029] Furthermore, the noise reduction method for reducing noise of all the component speech signals includes:

[0030] Acquire a filter allocation weight for each of the component speech segments according to the first filter weight and the second filter weight, wherein the first filter weight and the second filter weight are both positively correlated with the filter allocation weight;

[0031] The filter allocation weight of each of the component speech segments is multiplied and combined with the noise intensity of the classroom speech signal to obtain the filter intensity of each of the component speech segments;

[0032] The filtering strength is used as a filtering parameter of a filter to perform noise reduction on all the component speech signals.

[0033] Furthermore, the method for acquiring the component voice signal includes:

[0034] The classroom speech signal is decomposed by using the EMD algorithm to obtain all component speech signals.

[0035] Furthermore, the method for obtaining classroom speech segments includes:

[0036] The classroom speech signal is evenly divided into a preset number of segments to obtain all classroom speech segments.

[0037] The present invention proposes a multimodal classroom teacher teaching speech behavior analysis system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multimodal classroom teacher teaching speech behavior analysis method are implemented.

[0038] The present invention has the following beneficial effects:

[0039] The present invention first obtains a classroom voice signal and all component voice signals of the classroom voice signal; segments and numbers the classroom voice signal and all component voice signals, and the lengths of the classroom voice segments and the component voice segments with the same serial number are consistent, so as to analyze the characteristics between the classroom voice segments and the component voice segments in the same time period, thereby evaluating the interference degree of each voice segment, and adaptively adjusting the filtering strength for noise reduction; according to the distribution feature changes of the component voice segments and the classroom voice segments in each component voice signal, the change contribution of each component voice segment is obtained, and the change contribution reflects the similarity of the distribution feature changes of the component voice segments and the classroom voice segments in each component voice signal, and then the noise interference can be evaluated; according to the difference in the change contribution between different component voice segments under the same serial number, combined with each component voice segment, the change contribution of each component voice segment is obtained. The contribution rate of each component speech segment is obtained, and the first filtering weight of each component speech segment is obtained; according to the similarity of the spectrum between different component speech segments of each component speech signal and the consistency of the time-frequency distribution of each component speech segment, the second filtering weight of each component speech segment is obtained; according to the first filtering weight and the second filtering weight, combined with the noise intensity of the classroom speech signal, all component speech signals are denoised, the noise intensity of the classroom speech signal reflects the overall situation of its noise interference, the first filtering weight and the second filtering weight are based on the characteristics of each component speech segment to analyze the possibility of different types or intensities of noise interference it suffers, combined with the first filtering weight and the second filtering weight, each component speech segment can be adaptively filtered to different degrees, thereby improving the filtering effect; according to all the component speech signals after denoising, teaching speech behavior analysis is performed. The present invention analyzes the classroom speech signal and all its component speech signals in segments, evaluates the influence of noise interference on each segment to determine the first filtering weight and the second filtering weight, and then adaptively filters each segment to improve the noise reduction effect of the classroom speech signal, thereby improving the speech behavior analysis effect of the teacher's classroom teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flowchart of a method for analyzing a multimodal classroom teacher's teaching speech behavior provided by an embodiment of the present invention;

[0042] Figure 2 A waveform diagram of a classroom speech signal provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a classroom speech signal and all component speech signals provided by an embodiment of the present invention;

[0044] Figure 4 A waveform diagram of a reconstructed classroom speech signal after noise reduction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a multimodal classroom teacher teaching speech behavior analysis method and system proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0046] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0047] The specific scheme of a multimodal classroom teacher teaching speech behavior analysis method and system provided by the present invention is described in detail below with reference to the accompanying drawings.

[0048] See also Figure 1 , which shows a flowchart of a method for analyzing a multimodal classroom teacher's teaching speech behavior provided by an embodiment of the present invention, specifically comprising:

[0049] Step S1, obtaining a classroom voice signal and all component voice signals of the classroom voice signal; segmenting and numbering the classroom voice signal and all component voice signals to obtain all classroom voice segments and component voice segments, and the lengths of classroom voice segments and component voice segments with the same serial number are consistent.

[0050] In one embodiment of the present invention, in order to improve the effect of analyzing the teacher's speech behavior during classroom teaching, firstly, an audio acquisition device, such as a microphone or a recording device, is installed at the classroom podium to capture the teacher's voice information during the teaching process, and then the collected voice information is converted into classroom voice signals for subsequent decomposition and noise reduction; please refer to Figure 2 , which shows a waveform diagram of a classroom speech signal provided by an embodiment of the present invention; then the classroom speech signal is decomposed using the EMD algorithm to obtain multimodal components to obtain all component speech signals.

[0051] It should be noted that in other embodiments of the present invention, the implementer may also arrange the audio acquisition equipment at different locations in the classroom to form a microphone array or multiple channel recording, and then use array signal processing technology or speech synthesis technology to synthesize the multiple voice signals to ensure that comprehensive voice data is captured; this and the acquisition of voice signals and EMD algorithms are existing technologies well known to those skilled in the art and will not be elaborated here.

[0052] Taking into account that external interference such as keyboard tapping or noise outside the teacher is usually randomly distributed in the classroom voice signal, different voice segments may be affected by noise of different types and intensities. Directly decomposing and denoising the entire voice classroom signal may affect the final speech behavior analysis effect; therefore, the embodiment of the present invention segments and numbers the classroom voice signal and the multimodal component voice signal to obtain all classroom voice segments and component voice segments, wherein the lengths of classroom voice segments and component voice segments with the same serial number are consistent, so as to analyze the characteristics between classroom voice segments and component voice segments in the same time period, thereby evaluating the degree of interference of each voice segment, and adaptively adjusting the filtering strength for noise reduction.

[0053] In one embodiment of the present invention, the classroom speech signal is divided into a preset number of segments to obtain all classroom speech segments. The preset number is 10, and the implementer can set the value according to the specific implementation method, which is not limited here.

[0054] To understand the segmentation process of the classroom speech signal and all its component speech signals, please refer to Figure 3 , which shows a segmented schematic diagram of a classroom speech signal and all component speech signals provided by an embodiment of the present invention; Figure 3 In , classroom speech data represents the collected classroom speech signal, r1 represents the first-order component speech signal of the classroom speech signal, r2 represents the second-order component speech signal of the classroom speech signal, and r3 represents the third-order component speech signal of the classroom speech signal. By aligning the classroom speech signal with the first signal point of all component speech signals and then evenly segmenting the classroom speech signal, multiple classroom speech segments of equal length can be obtained, and then component speech segments of all component speech signals can be obtained vertically, where d1 represents the classroom speech signal and the first speech segment of each component speech signal, d2, d3 and d4 and so on, and the lengths of all speech segments in the vertical direction, i.e., the same segment number, are consistent.

[0055] It should be noted that Figure 3It is only a segmentation method to help understand the classroom speech signal and all component speech signals. It does not mean that the classroom speech signal only has third-order component speech signals, nor does it mean that the classroom speech segment is divided into 4 segments. In other embodiments of the present invention, the implementer can adopt other segmentation methods at his own will, such as unequally dividing the classroom speech signal into multiple segments, but it is necessary to ensure that the lengths of all classroom speech segments and component speech segments with the same segment number in the vertical direction are consistent for subsequent analysis. After the signal is decomposed, the total length of the classroom speech signal and all component speech signals may not be consistent, resulting in the possible inconsistency of the lengths of all speech segments of the last segment number, which does not affect the implementation of this solution. The implementer can also interpolate or extend all segments of the last segment number to align the lengths, which is not limited or elaborated here.

[0056] Step S2, according to the changes in the distribution characteristics of the component speech segments and the classroom speech segments in each component speech signal, obtain the change contribution of each component speech segment; according to the difference in the change contribution between different component speech segments under the same sequence number, combined with the contribution rate of each component speech segment in all component speech segments under the same sequence number, obtain the first filtering weight of each component speech segment; according to the time-frequency distribution characteristics between different component speech segments of each component speech signal, obtain the second filtering weight of each component speech segment; according to the first filtering weight and the second filtering weight, combined with the noise intensity of the classroom speech signal, denoise all component speech signals.

[0057] Different speech segments may be affected by different types and intensities of noise, that is, the distribution characteristics of signal points in different speech segments of the classroom speech signal or each component speech signal may not be consistent; and considering that if the distribution characteristic changes of the component speech segment and the classroom speech segment in each component speech signal are relatively similar, it means that the corresponding component speech segment better retains the signal structure and characteristics of the corresponding classroom speech segment, and the degree of noise interference in the component speech segment may be low. Therefore, the embodiment of the present invention obtains the change contribution of each component speech segment based on the distribution characteristic changes of the component speech segment and the classroom speech segment in each component speech signal. The change contribution reflects the similarity of the distribution characteristic changes of the component speech segment and the classroom speech segment in each component speech signal, and then the noise interference can be evaluated.

[0058] Preferably, in one embodiment of the present invention, considering that the frequency resolution in the signal reflects the distribution detail characteristics of the frequency components in a signal, it can be judged whether its frequency components are easy to distinguish, and the distribution disorder of the signal points also reflects the corresponding signal distribution characteristics, and the distribution characteristic parameters can be constructed by combining the two distribution characteristics; considering that the ratio of the change amount of the distribution characteristic parameters between adjacent component speech segments and the change amount of the distribution characteristic parameters between adjacent classroom speech segments can reflect the similarity of the distribution characteristic changes; based on this, the method for obtaining the change contribution includes:

[0059] According to the frequency resolution of each speech segment and the distribution disorder of the signal points, the distribution characteristic parameters of each speech segment are obtained;

[0060] In each component speech signal, taking the difference in distribution characteristic parameters between each component speech segment and the next adjacent component speech segment as a first parameter difference;

[0061] In the classroom speech signal, the difference in distribution characteristic parameters between each classroom speech segment and the next adjacent classroom speech segment is used as the second parameter difference;

[0062] Between corresponding speech segments with the same sequence number, the ratio of the first parameter difference to the second parameter difference is used as the change contribution of the corresponding component speech segment.

[0063] It should be noted that, if the last speech segment of the classroom speech signal and each component speech signal has no next adjacent speech segment, its change contribution will not be calculated, and it will not be filtered in the subsequent processing process; in other embodiments of the present invention, the implementer may also make the change contribution of the last speech segment equal to the change contribution of its adjacent previous speech segment, or extend the classroom speech signal and each component speech signal to obtain the next vector segment of the last speech segment, and then obtain the change contribution of the last speech segment.

[0064] The calculation formula for change contribution is:

[0065] ;

[0066] in, is the sequence number of the component speech signal; It is the serial number of the component speech segment or the classroom speech segment; For the The first component of the speech signal The contribution of each component speech segment to the change; For the The first component of the speech signal Distribution characteristic parameters of the component speech segments; For the The first component of the speech signal Distribution characteristic parameters of the component speech segments; The first Distribution characteristic parameters of classroom speech segments; The first Distribution characteristic parameters of classroom speech segments; is the absolute value symbol.

[0067] In the calculation formula of the change contribution, the closer the ratio is to 1, the more similar the distribution characteristic changes between adjacent component speech segments are to the distribution characteristic changes between adjacent classroom speech segments, and the lower the possibility of the corresponding component speech segment being disturbed by noise.

[0068] Preferably, in one embodiment of the present invention, considering that information entropy can reflect the degree of distribution disorder of signal points in speech segments, and considering that the time interval between adjacent extreme value points can reflect the density distribution of frequency components in the signal, the larger the time interval, the greater the ability to distinguish the frequency components; based on this, the method for obtaining characteristic parameters includes:

[0069] Obtain the information entropy of each speech segment;

[0070] Obtain the extreme value points in each speech segment; in each speech segment, calculate the time intervals between adjacent extreme value points, and use the average of all time intervals as the frequency resolution of the speech segment;

[0071] According to the information entropy and frequency resolution of each speech segment, the distribution characteristic parameters of each speech segment are obtained. The information entropy and frequency resolution are positively correlated with the distribution characteristic parameters.

[0072] The calculation formula of distribution characteristic parameters is:

[0073] ;

[0074] in, is the serial number of the component speech segment or the classroom speech segment, that is, the serial number of the speech segment; For the Distribution characteristic parameters of speech segments; is the amplitude category serial number of the signal point of the same amplitude category in the speech segment; is the total number of amplitude categories of all signal points of the same amplitude category in the speech segment; For the speech segment The total number of signal points corresponding to the amplitude category; is the total number of signal points in the speech segment; is a logarithmic function; Information entropy for speech segmentation; is the combination serial number of adjacent extreme value points; is the total number of combinations of adjacent extreme points, that is, the total number of extreme points minus 1; For the speech segment The collection time corresponding to the extreme point; For the speech segment The collection time corresponding to the extreme point; For the In the combination of adjacent extreme points, the The extreme point and The time interval between extreme points; The frequency resolution at which the speech is segmented.

[0075] It should be noted that in the embodiment of the present invention, the information entropy and frequency resolution of the speech segment are combined by multiplication to represent a positive correlation. In other embodiments of the present invention, the implementer may also use basic mathematical operations such as addition, weighted summation or exponential power to represent a positive correlation, which will not be elaborated or limited here. The method for obtaining extreme value points can be obtained by fitting the signal curve function and then deriving it, or by using algorithms such as the local extreme value detection algorithm. It and the method for obtaining information entropy are both existing technical means well known to those skilled in the art and will not be elaborated here.

[0076] It should be noted that the distribution characteristic parameter only represents the distribution characteristics of the speech segment, so as to analyze the contribution rate of the change; the larger the distribution characteristic parameter value, the more obvious the corresponding signal distribution characteristics.

[0077] In another embodiment of the present invention, the implementer may also obtain the variance of the signal amplitude of each speech segment, and perform spectrum analysis on each speech segment to obtain its spectrum density; after positive correlation merging the variance and spectrum density, the distribution characteristic parameters of the corresponding speech segment are obtained. Variance and spectrum density are both prior arts well known to those skilled in the art, and will not be described in detail here; in other embodiments of the present invention, the implementer may also obtain other types of parameters such as kurtosis and power density that can represent the distribution characteristics of the speech segment, and further obtain the distribution characteristic parameters.

[0078] Taking into account that the more important a component speech segment is, the greater its importance for the subsequent analysis of the teacher's teaching speech behavior, and the higher the necessity of filtering and denoising it, in order to obtain accurate speech data for analysis; and considering that when the distribution feature changes of a component speech segment are different from the distribution feature changes of other component speech segments with the same sequence number, that is, in the same time period, it is indirectly indicated that the greater the possibility of noise interference in the component speech segment, the greater its filtering weight should be; therefore, the embodiment of the present invention obtains the first filtering weight of each component speech segment according to the difference in the contribution of changes between different component speech segments with the same sequence number, combined with the importance of each component speech segment.

[0079] Preferably, in one embodiment of the present invention, considering that when a component speech segment is removed and reconstructed, the reconstructed speech segment changes greatly relative to the corresponding classroom speech segment, it means that the contribution rate of the component speech segment is high, and the higher the contribution rate, the higher the importance of the component speech segment; and considering that when the contribution degree of change of a component speech segment is greatly different from the contribution degrees of change of other component speech segments with the same sequence number, that is, in the same time period, it is indirectly explained that the component speech segment is more likely to be interfered by noise, and it is more likely to be an abnormal segment that is seriously interfered by noise; based on this, the method for obtaining the first filtering weight includes:

[0080] Taking any component speech segment as the target segment, obtaining the reconstructed speech segment of all component speech segments except the target segment under the same sequence number, and obtaining the distribution characteristic parameters of the reconstructed speech segment; taking the difference in the distribution characteristic parameters between the reconstructed speech segment and the classroom speech segment under the same sequence number as the importance parameter of the target segment;

[0081] Under the same sequence number, the average value of the difference in the change contribution between the target segment and all other component speech segments is obtained, and the average value is used as the abnormal parameter of the target segment;

[0082] According to the importance parameter and the abnormality parameter of each component speech segment, the first filtering weight of each component speech segment is obtained, and the importance parameter and the abnormality parameter are both positively correlated with the first filtering weight.

[0083] The calculation formula of the first filter weight is:

[0084] ;

[0085] in, For the The first component of the speech signal first filter weights for the component speech segments; For the Distribution characteristic parameters of speech segments; For the Under each segment number, except The distribution characteristic parameters of the reconstructed speech segments of all the component speech segments other than the component speech signal; is the absolute value symbol; For the The first component of the speech signal Importance parameter of each component speech segment; For the The sequence numbers of the remaining component voice signals other than the component voice signal; For the The total number of component voice signals other than the component voice signal, that is, the total number of component voice signals minus 1; For the The first component of the speech signal The contribution of each component speech segment to the change; For the The first component of the speech signal The contribution of each component speech segment to the change; For the Abnormal parameters of the component speech signals.

[0086] It should be noted that in other embodiments of the present invention, the implementer may also evaluate the contribution rate of each component speech signal by calculating the energy of each component speech signal and the energy of the classroom speech signal to obtain the importance parameter of each component speech segment; or the contribution rate of each component speech signal may be evaluated by the mean square error of the reconstructed signal segment and the classroom speech segment after eliminating each component speech segment. These are all existing technologies well known to those skilled in the art and will not be elaborated here.

[0087] In the calculation formula of the first filter weight, the importance parameter and the abnormal parameter of each component speech segment are multiplied and combined. In other embodiments of the present invention, the two can also be combined by adding or exponential idempotent basic mathematical operations so that they are both positively correlated with the first filter weight. These are all existing technologies well known to those skilled in the art and will not be elaborated here.

[0088] Considering that when the spectral characteristics of all component speech segments in a component speech signal are relatively similar, it means that the component speech signal has a certain periodicity, and the interference degree of each component speech segment is relatively consistent, it may be difficult to distinguish the interference degree of each component speech segment; considering that the greater the difference in the distribution characteristics of each component speech segment in the time domain and the frequency domain, the poorer the consistency of the component speech segment under different forms of expression, which further indirectly reflects that the component speech segment may be affected by noise interference; therefore, the embodiment of the present invention obtains the second filtering weight of each component speech segment according to the spectral similarity between different component speech segments of each component speech signal and the consistency of the time-frequency distribution of each component speech segment.

[0089] Preferably, in one embodiment of the present invention, the difference in distribution probability of signal points at the same sequence number position in the spectrum signal corresponding to different component speech segments in the same component speech signal is taken into account, which indirectly reflects the similarity between the corresponding component speech segments; and the signal amplitude distribution of the component speech segments in the time domain is taken into account;

[0090] Based on this, the method for obtaining the second filtering weight includes:

[0091] Performing Fourier transformation on each component speech segment to obtain a frequency domain component speech segment of each component speech segment; obtaining a segment similarity parameter of each component speech signal according to the distribution probability difference between signal points with the same sequence number in different frequency domain component speech segments;

[0092] The amplitude variance of the signal point in each component speech segment is used as the time domain amplitude variance, and the amplitude of each signal point in the component speech segment is weighted and averaged according to the time domain amplitude variance to obtain the time domain feature parameter of each component speech segment; the amplitude variance of the signal point in each frequency domain component speech segment is used as the frequency domain amplitude variance, and the amplitude of each signal point in the frequency domain component speech segment is weighted and averaged according to the frequency domain amplitude variance to obtain the frequency domain feature parameter of each component speech segment;

[0093] The difference between the time domain characteristic parameter and the frequency domain characteristic parameter of each component speech segment is used as the time-frequency distribution consistency parameter of each component speech segment;

[0094] According to the segmented similarity parameter and the time-frequency distribution consistent parameter, a second filtering weight of each component speech signal is obtained, and both the segmented similarity parameter and the time-frequency distribution consistent parameter are positively correlated with the second filtering weight.

[0095] The calculation formula of the second filter weight is:

[0096] ;

[0097] in, For the The first component of the speech signal second filter weights for the component speech segments; For the The piecewise similarity parameters of the component speech signals; is the sequence number of the signal point in the component speech segment; is the total number of signal points in the component speech segment; The first The amplitude of the signal point; is the amplitude variance of all signal points in the component speech segment, i.e., the time domain amplitude variance; is the serial number of the signal point in the frequency domain component speech segment; is the total number of signal points in the speech segment of the frequency domain component; is the frequency domain component speech segment The amplitude of the signal point; is the amplitude variance of all signal points in the frequency domain component speech segment, that is, the frequency domain amplitude variance; is the absolute value symbol; For the The first component of the speech signal The time domain characteristic parameters of the component speech segments; For the The first component of the speech signal The frequency domain feature parameters of the component speech segments.

[0098] In the calculation formula of the second filter weight, Represents the time-frequency distribution consistency parameter; the segmented similarity parameter of each component speech signal and the time-frequency distribution consistency parameter of each component speech segment in each component speech signal are combined by multiplication. The smaller the segmented similarity parameter is, the greater the similarity between the segments is, and the more difficult it is to distinguish the degree of interference of each component speech segment. The corresponding second filtering weight should be set lower, and then the second filtering weight is set in combination with the time-frequency distribution consistency parameter of each component speech segment to avoid excessive noise reduction causing information loss of the corresponding component speech signal; when the time-frequency distribution consistency parameter of each component speech segment is large, it means that the time-frequency distribution difference is large, then the component speech segment may be greatly affected by noise interference, and the corresponding second filtering weight is larger. In other embodiments of the present invention, the exponential idempotent basic mathematical operation can also be used to merge the two so that they are both positively correlated with the second filtering weight. They are all prior arts well known to those skilled in the art and will not be described here.

[0099] The calculation formula of the piecewise similarity parameter includes:

[0100] ;

[0101] in, For the The piecewise similarity parameters of the component speech signals; For the The combined serial number of the frequency domain component speech segments corresponding to any two component speech segments in the component speech signal; For the The total number of combinations of frequency domain component speech segments corresponding to any two component speech segments of a component speech signal; is the serial number of the signal point in each frequency domain component speech segment; is the total number of signal points in the speech segment for each frequency domain component; For the The first frequency domain component speech segment The amplitude of the signal point; For the The first frequency domain component speech segment The amplitude of the signal point; is a logarithmic function.

[0102] In the calculation formula of the piecewise similarity parameter, The difference between the corresponding distribution probabilities of signal points with the same serial number in different frequency domain component speech segments is evaluated in the form of analog relative entropy. The smaller the difference, the higher the similarity of the spectral features between the two corresponding frequency domain component speech segments. By averaging the similarities of the spectral features between all frequency domain component speech segments, the segmental similarity of each component speech signal can be evaluated. The higher the segmental similarity, the smaller the corresponding segmental similarity parameter.

[0103] Considering that the first filter weight and the second filter weight are obtained by analyzing the possibility of different types or intensities of noise interference to each component speech segment based on the characteristics of each component speech segment, and then adaptively filtering each component speech segment to different degrees according to the noise interference of each component speech segment. Therefore, after obtaining the first filter weight and the second filter weight of each component speech segment, the embodiment of the present invention combines the noise intensity of the classroom speech signal to reduce the noise of all component speech signals.

[0104] Preferably, in one embodiment of the present invention, the noise reduction method for all component speech signals includes:

[0105] Obtaining a filter allocation weight for each component speech segment according to the first filter weight and the second filter weight, wherein the first filter weight and the second filter weight are both positively correlated with the filter allocation weight;

[0106] The filter allocation weight of each component speech segment is multiplied and combined with the noise intensity of the classroom speech signal to obtain the filter intensity of each component speech segment;

[0107] The filtering strength is used as the filtering parameter of the filter to reduce the noise of all component speech signals.

[0108] In one embodiment of the present invention, the specific calculation formula of the filtering allocation weight is: ;in, For the The first component of the speech signal Filter allocation weights for each component speech segment; For the The first component of the speech signal first filter weights for the component speech segments; For the The first component of the speech signal In other embodiments of the present invention, the implementer may also combine the two by using basic mathematical operations such as addition or weighted summation, which are well known to those skilled in the art and will not be described in detail here.

[0109] It should be noted that the acquisition of the noise intensity of the classroom speech signal and the setting of the filter parameters according to the filtering intensity are prior arts well known to those skilled in the art; in one embodiment of the present invention, the method for acquiring the noise intensity of the classroom speech signal is specifically as follows: first, the energy of each component speech signal of the classroom speech signal is acquired, the energy change rate of the adjacent order component speech signal is acquired, and all energy change rates are averaged; each component speech signal is subjected to Fourier transform to acquire its main frequency, the change rate of the main frequency of the adjacent order component speech signal is acquired, and all the change rates of the main frequencies are averaged; the mean of all energy change rates and the mean of the change rates of all main frequencies are weighted and averaged, wherein the corresponding weight of the mean of all energy change rates is 0.6, and the corresponding weight of the mean of the change rates of all main frequencies is 0.3, and the weighted average is used as the noise intensity. In other embodiments of the present invention, the implementer may adopt other noise intensity estimation methods to acquire the noise intensity of the classroom speech signal.

[0110] In one embodiment of the present invention, the filter is a Gaussian filter, and the filtering strength of each component speech segment is used as the standard deviation that needs to be set when the Gaussian filter is filtered. All component speech segments are filtered, and then the filtered component speech segments are spliced ​​to obtain corresponding component speech signals.

[0111] Step S3, performing teaching speech behavior analysis based on the noise reduction result.

[0112] In one embodiment of the present invention, all the component speech signals after noise reduction are superimposed and reconstructed to obtain a high-quality reconstructed classroom speech signal. Figure 4, which shows a waveform diagram of a reconstructed classroom speech signal after noise reduction provided by an embodiment of the present invention. After obtaining the reconstructed classroom speech signal, speech behavior analysis can be performed on the reconstructed classroom speech signal; the teacher's teaching speech behavior is evaluated by analyzing the timbre, pitch, speech speed, emotion, semantic content and silence duration of the reconstructed classroom speech signal. Speech behavior analysis is a prior art well known to those skilled in the art and will not be elaborated here.

[0113] In summary, the embodiment of the present invention first obtains the classroom voice signal and all component voice signals of the classroom voice signal; segments and numbers the classroom voice signal and all component voice signals; further obtains the change contribution of each component voice segment; further obtains the first filtering weight of each component voice segment; further obtains the second filtering weight of each component voice segment; according to the first filtering weight and the second filtering weight, combined with the noise intensity of the classroom voice signal, all component voice signals are denoised, thereby performing speech behavior analysis. The present invention analyzes the classroom voice signal and all its component voice signals in segments, evaluates the impact of noise interference on each segment to determine the first filtering weight and the second filtering weight, and then performs adaptive filtering on each segment to improve the noise reduction effect of the classroom voice signal, thereby improving the speech behavior analysis effect of the teacher's classroom teaching.

[0114] The present invention also proposes a multimodal classroom teacher teaching speech behavior analysis system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a multimodal classroom teacher teaching speech behavior analysis method.

[0115] A method and system embodiment of multimodal classroom teacher teaching speech noise reduction:

[0116] The voice data of teachers in the classroom teaching process contains rich information, including speech speed, intonation, emotional expression and other information that can be used to analyze the teacher's speech behavior, analyze and identify the teacher's language characteristics and emotional expression, such as whether the teacher has emotional fluctuations during the teaching process, whether the language expression is vivid and interesting to attract students' interest in learning, etc. This helps to have a deeper understanding of the teacher's speech behavior and provide a scientific basis for teaching evaluation and improvement. However, in the actual teaching environment, there will be various background interferences, such as air conditioning noise, keyboard tapping, student voices, etc. These interferences will reduce the accuracy of the collection and analysis of teacher voice data. However, the interference factors are randomly distributed, and conventional noise removal methods cannot distinguish between actual voice information and interference information well. The present invention proposes a multimodal classroom teacher teaching speech noise reduction method, which specifically includes:

[0117] Step S1, obtaining a classroom voice signal and all component voice signals of the classroom voice signal; segmenting and numbering the classroom voice signal and all component voice signals to obtain all classroom voice segments and component voice segments, and the lengths of classroom voice segments and component voice segments with the same serial number are consistent.

[0118] Step S2, according to the changes in the distribution characteristics of the component speech segments and the classroom speech segments in each component speech signal, obtain the change contribution of each component speech segment; according to the difference in the change contribution between different component speech segments under the same sequence number, combined with the contribution rate of each component speech segment in all component speech segments under the same sequence number, obtain the first filtering weight of each component speech segment; according to the time-frequency distribution characteristics between different component speech segments of each component speech signal, obtain the second filtering weight of each component speech segment; according to the first filtering weight and the second filtering weight, combined with the noise intensity of the classroom speech signal, denoise all component speech signals.

[0119] Among them, steps S1 to S2 have been described in detail in the above-mentioned embodiment of a multimodal classroom teacher teaching speech behavior analysis method, and will not be repeated here.

[0120] The present invention first obtains a classroom voice signal and all component voice signals of the classroom voice signal; segments and numbers the classroom voice signal and all component voice signals; further obtains the change contribution of each component voice segment; further obtains the first filter weight of each component voice segment; further obtains the second filter weight of each component voice segment; according to the first filter weight and the second filter weight, combined with the noise intensity of the classroom voice signal, all component voice signals are denoised. The present invention performs segmented analysis on the classroom voice signal and all its component voice signals, evaluates the impact of noise interference on each segment to determine the first filter weight and the second filter weight, and then performs adaptive filtering on each segment to improve the noise reduction effect of the classroom voice signal.

[0121] The present invention also proposes a multimodal classroom teacher teaching speech noise reduction system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a multimodal classroom teacher teaching speech noise reduction method are implemented.

[0122] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0123] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for analyzing teachers’ teaching speech behavior in multimodal classrooms. It is characterized in that The method comprises: Acquire a classroom voice signal and all component voice signals of the classroom voice signal; segment and number the classroom voice signal and all the component voice signals to obtain all classroom voice segments and component voice segments, wherein the classroom voice segments and component voice segments with the same sequence number have the same length; According to the changes in the distribution characteristics of the component speech segments and the classroom speech segments in each of the component speech signals, the change contribution of each of the component speech segments is obtained; according to the difference in the change contribution between different component speech segments under the same sequence number, combined with the contribution rate of each component speech segment in all the component speech segments under the same sequence number, the first filtering weight of each component speech segment is obtained; according to the time-frequency distribution characteristics of each component speech segment of each component speech signal, the second filtering weight of each component speech segment is obtained; according to the first filtering weight and the second filtering weight, combined with the noise intensity of the classroom speech signal, all the component speech signals are denoised; Conduct teaching speech behavior analysis based on the noise reduction results; The calculation formula of the change contribution is: ; in, is the sequence number of the component speech signal; It is the serial number of the component speech segment or the classroom speech segment; For the The first component of the speech signal The contribution of each component speech segment to the change; For the The first component of the speech signal Distribution characteristic parameters of component speech segments; For the The first component of the speech signal Distribution characteristic parameters of component speech segments; The first Distribution characteristic parameters of classroom speech segments; The first Distribution characteristic parameters of classroom speech segments; is the absolute value symbol.

2. According to claim 1, a multimodal classroom teacher teaching speech behavior analysis method, It is characterized in that The method for obtaining the distribution characteristic parameters includes: Obtain the information entropy of each speech segment; Obtaining extreme value points in each speech segment; in each speech segment, calculating the time intervals between adjacent extreme value points, and taking the average of all the time intervals as the frequency resolution of the speech segment; According to the information entropy and the frequency resolution of each speech segment, the distribution characteristic parameters of each speech segment are obtained, and the information entropy and the frequency resolution are both positively correlated with the distribution characteristic parameters.

3. According to claim 1, a multimodal classroom teacher teaching speech behavior analysis method, It is characterized in that The method for obtaining the first filtering weight includes: Taking any of the component speech segments as a target segment, obtaining reconstructed speech segments of all the component speech segments except the target segment under the same sequence number, and taking the difference in the distribution feature parameters between the reconstructed speech segment and the classroom speech segment under the same sequence number as the importance parameter of the target segment; Under the same sequence number, obtaining an average value of the difference in the change contribution between the target segment and all the other component speech segments, and taking the average value as the abnormal parameter of the target segment; According to the importance parameter and the abnormal parameter of each of the component speech segments, a first filtering weight of each of the component speech segments is obtained, and both the importance parameter and the abnormal parameter are positively correlated with the first filtering weight.

4. According to claim 1, a multimodal classroom teacher teaching speech behavior analysis method, It is characterized in that The method for acquiring the second filtering weight includes: Performing Fourier transformation on each of the component speech segments to obtain a frequency domain component speech segment of each of the component speech segments; obtaining a segment similarity parameter of each of the component speech signals according to a distribution probability difference between signal points with the same sequence number in different frequency domain component speech segments; The amplitude variance of the signal point in each of the component speech segments is used as the time domain amplitude variance, and the amplitude of each signal point in the component speech segment is weighted and averaged according to the time domain amplitude variance to obtain the time domain feature parameter of each of the component speech segments; the amplitude variance of the signal point in each of the frequency domain component speech segments is used as the frequency domain amplitude variance, and the amplitude of each signal point in the frequency domain component speech segment is weighted and averaged according to the frequency domain amplitude variance to obtain the frequency domain feature parameter of each of the component speech segments; Taking the difference between the time domain feature parameter and the frequency domain feature parameter of each of the component speech segments as the time-frequency distribution consistency parameter of each of the component speech segments; According to the segmented similarity parameter and the time-frequency distribution consistency parameter, a second filtering weight of each component speech signal is obtained, and the segmented similarity parameter and the time-frequency distribution consistency parameter are both positively correlated with the second filtering weight.

5. A method for analyzing multimodal classroom teacher teaching speech behavior according to claim 4, It is characterized in that The calculation formula of the segment similarity parameter includes: ;in, For the The piecewise similarity parameters of the component speech signals; For the The combined serial number of the frequency domain component speech segments corresponding to any two component speech segments in the component speech signal; For the The total number of combinations of frequency domain component speech segments corresponding to any two component speech segments in the component speech signal; is the serial number of the signal point in each frequency domain component speech segment; is the total number of signal points in the speech segment for each frequency domain component; For the The first frequency domain component speech segment The amplitude of the signal point; For the The first frequency domain component speech segment The amplitude of the signal point; is a logarithmic function.

6. A method for analyzing the teaching speech behavior of a multimodal classroom teacher according to claim 1, It is characterized in that The noise reduction method for reducing noise of all the component speech signals includes: Acquire a filter allocation weight for each of the component speech segments according to the first filter weight and the second filter weight, wherein the first filter weight and the second filter weight are both positively correlated with the filter allocation weight; The filter allocation weight of each of the component speech segments is multiplied and combined with the noise intensity of the classroom speech signal to obtain the filter intensity of each of the component speech segments; The filtering strength is used as a filtering parameter of a filter to perform noise reduction on all the component speech signals.

7. A method for analyzing the teaching speech behavior of a multimodal classroom teacher according to claim 1, It is characterized in that The method for acquiring the component voice signal comprises: The classroom speech signal is decomposed by using the EMD algorithm to obtain all component speech signals.

8. According to claim 1, a method for analyzing the teaching speech behavior of a multimodal classroom teacher, It is characterized in that The method for obtaining classroom speech segments includes: The classroom speech signal is evenly divided into a preset number of segments to obtain all classroom speech segments.

9. A multimodal classroom teacher teaching speech behavior analysis system, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for analyzing the teaching speech behavior of a multimodal classroom teacher are implemented as described in any one of claims 1 to 8.