Audio and video online learning method based on blackboard model cooperation

Abstract

The invention discloses an audio and video online learning method based on blackboard model collaboration, relates to the technical field of blackboard models, and solves the problem that the association efficiency is too slow during actual process processing and content output. Classifying the contents of the same type into one class, and setting a division mark; the clear division mode is beneficial to carrying out targeted processing and management on different types of contents, and convenience is provided for subsequent type combination and index output; intelligent and efficient type combination: according to the relationship between the total number of index paths and the total number of type contents, intelligently determining whether to carry out type combination, and selecting an optimal process by calculating a process variance; according to the optimal combination mode, resources can be reasonably allocated, so that different types of contents can be efficiently processed under a limited index path, the waste of the resources is avoided, and the overall index efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of blackboard models, and in particular to an audio and video online learning method based on blackboard model collaboration. Background Art

[0002] With the rapid development of the times, the intelligent blackboard model has gradually emerged. The model selects relevant text content from the cloud database based on the voice content of the corresponding teacher during class, and outputs it to facilitate the corresponding teaching content, etc. However, when the intelligent blackboard model is performing collaborative processing, it not only needs to process the collected audio and video content in real time, but also needs to select the relevant content based on the processed feature text and output the selected content. In the actual processing process, the processing process is relatively slow, which will result in the output efficiency being too slow when the content is output, resulting in users not being able to get a good user experience. Summary of the invention

[0003] In view of the deficiencies of the prior art, the present invention provides an audio and video online learning method based on blackboard model collaboration, which solves the problem that the efficiency associated with the actual processing process and content output is too slow.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: an audio and video online learning method based on blackboard model collaboration, comprising the following steps: Step 1: Acquire the audio signal generated during the class, and generate a specific audio feature text based on the relevant features of the acquired audio signal. The specific sub-steps are: S11, collecting audio signals according to standard audio sampling parameters, with the sampling bit number being 16 bits, dividing the collected single audio signal into short frames, so that the single audio signal generates a plurality of short frames, and generating a short frame sequence; S12, extracting features from each short frame in the short frame sequence, and confirming the audio comprehensive features of the audio signal based on the MFCC feature value associated with each short frame: S121, perform FFT on each short frame, convert the time domain signal into a frequency domain signal, and obtain a spectrum; the formula of FFT is: , where X[k] is the frequency domain signal, x[n] is the time domain signal, N is the length of the frame, n is the index of the time domain signal, indicating the time point, k is the index of the frequency domain signal, indicating the frequency point, and j is the imaginary unit, indicating the complex exponential function; S122. Then square the FFT result to get the power spectrum: P[k]=|X[k]| 2 ; S123, the power spectrum is passed through a set of Mel filters to convert the linear frequency into Mel frequency. The conversion formula of Mel frequency is: , where f is the linear frequency, f mel is the Mel frequency; S124, then take the logarithm of the output of each Mel filter to obtain the logarithmic energy: , where H m [k] is the frequency response of the mth Mel filter, and E[m] is the logarithmic energy; S125, performing discrete cosine transform on the logarithmic energy to obtain MFCC coefficients, wherein the formula of discrete cosine transform is: , where c[n] is the nth MFCC coefficient, M is the number of Mel filters, and the first 12-13 MFCC coefficients are averaged to determine the MFCC feature value of this short frame; S126, using the same processing steps as steps S121-S125 for the several short frames in the short frame sequence to sequentially confirm the MFCC feature values ​​of the corresponding short frames, and then summing the MFCC feature values ​​of the several short frames to lock the audio comprehensive features of the audio signal; S13, based on the audio comprehensive features associated with the corresponding audio signal, checking the audio signal with a preset feature checklist, confirming the check characters associated with the audio signal, wherein the feature checklist is a preset list, and sorting the check characters associated with the continuous audio signals to generate the audio feature text of this stage, wherein the interval time between the continuous audio signals is less than 1 second; Step 2: Check the audio feature text with the cloud database, select the check text from the cloud database, confirm the position of the check text, and mark the confirmed position as the index position. The specific sub-steps are: S21, mark the confirmed audio feature text as the main text, and confirm the text titles of other verification texts from the cloud database, use the confirmed text titles as the secondary text, and select similarity texts from the determined groups of secondary texts: perform character verification on the secondary text and the main text, mark the same characters as the same type of characters, and record the total number of the same type of characters G, then mark the total number of characters of the secondary text as H1, and the total number of characters of the main text as H2, use: ZB1=G÷H1 and ZB2=G÷H2 to confirm the proportion values ​​of the same type of characters ZB1 and ZB2, then lock the average proportion based on ZB1 and ZB2, and mark the secondary text with the average proportion ≥60% as the similarity text; S22, confirming the storage location of the similarity text from the cloud database, and marking the confirmed storage location as the index location; Step 3: Based on the confirmed index position and the preset index path, the text content associated with the index position is divided into types, multiple groups of type content are confirmed, and multiple groups of type content are combined according to the total number of index paths, the combined content is confirmed, and the multiple combined content is integrated into a content group to be indexed, specifically in the following manner: S31, based on the text content associated with the index position, confirm the different types included in the corresponding text content, divide the different types of content, divide the same type of content into the same category, lock multiple groups of different types of type content, and set division marks during the division process; S32, confirm the total number of preset index paths and mark it as Zs, then confirm the total number of different types of type content and mark it as ZG, if Zs≥ZG, there is no need to perform type combination, directly mark the corresponding type content as combined content, and integrate multiple combined contents into a content group to be indexed, if Zs<ZG, then perform type random combination of multiple groups of different types of type content, and continue for several combination processes, in each combination process, the total number of multiple groups of combined content is always consistent with ZG, select the best process from each combination process, the specific method is: S321, confirm the data capacity of different combination contents in each combination process, and mark the confirmed different data capacity as R i , where i represents different combinations of content, and then the confirmed multiple groups of data capacity R i Perform variance processing to lock the process variance of this combined process; S322, confirming the process variances associated with each different combination of processes in turn, and selecting the smallest process variance from the confirmed different process variances, and marking the combination process associated with the smallest process variance as the best process; S33, marking the multiple groups of combined contents associated with the best process as content groups to be indexed; Step 4: According to the marked content group to be indexed, the combined content within the content group to be indexed is indexed and output according to the preset index path. Based on the index features generated during the indexing process, the combined content that is most suitable for indexing and outputting in the corresponding index path is determined, the determined combined content is marked as the execution content, and indexed and outputted. The specific sub-steps are as follows: S41. Based on the total number of index paths Zs, select part of the content from a single combined content, and divide the selected part of the content into Zs sub-contents, distribute the Zs sub-contents to the associated index paths in sequence, and output the content through the associated index paths, and record multiple groups of index rates of the corresponding index paths, average the multiple groups of index rates, determine the index average rate of the corresponding index path, and mark it as SY o , where o represents different index paths. For this combination, select SYo The index path associated with max is used as the execution path of this combined content, and this combined content is marked as the execution content, and the execution content is indexed and output according to this execution path; S42, when each group of execution paths exists in the index path, the total number thereof is reduced by 1, and the remaining combined contents are processed in the same manner as step S41, the execution paths associated with the corresponding combined contents and the determined execution contents are confirmed, and the determined execution contents are indexed and output according to the determined execution paths; According to this processing method, different combinations of content are processed in turn, and the content is indexed and quickly output. The indexed multiple groups of combined content are reorganized according to the set division marks, the text content is confirmed, and the confirmed text content is directly displayed on the blackboard model.

[0005] The present invention provides an audio and video online learning method based on blackboard model collaboration. Compared with the prior art, it has the following beneficial effects: The present invention fully considers the time domain and frequency domain characteristics of audio by dividing the audio into short frames and extracting MFCC feature values ​​in the processing of class audio signals, simulating the human ear's perception of sound. This sophisticated processing method can accurately capture the characteristics of audio signals, providing a solid foundation for subsequent text matching, making the locked corresponding voice content associated text more accurate, and helping learners to accurately obtain key course information; When comparing audio feature text with the cloud database, similar texts are screened out by calculating the proportion of similar characters and determining the average proportion. This scientific screening method can quickly locate content that is highly relevant to audio feature text from a large amount of text, reducing unnecessary search time and improving data matching efficiency. The text content associated with the index position is divided into types, the content of the same type is classified into one category, and the division marks are set; this clear division method is conducive to the targeted processing and management of different types of content, which provides convenience for subsequent type combination and index output; the type combination is intelligent and efficient: according to the relationship between the total number of index paths and the total number of type content, it is intelligently determined whether to combine types, and the best process is selected by calculating the process variance; this optimized combination method can reasonably allocate resources, so that different types of content can be efficiently processed under limited index paths, avoiding waste of resources and improving overall indexing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0007] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0008] See also Figure 1 , this application provides an audio and video online learning method based on blackboard model collaboration, including the following steps: Step 1: Acquire the audio signal generated during the class, and generate a specific audio feature text based on the relevant features of the acquired audio signal. Specifically, different audios are associated with different audio features, and then based on different audio features and preset associated texts, the specific text associated with the corresponding audio features is locked. The specific sub-steps for generating the specific audio feature text are: S11. Collect audio according to standard audio sampling parameters. Generally, the sampling frequency is set to 16kHz or 44.1kHz, and the sampling bit number is 16 bits to ensure audio quality. The collected single audio signal is divided into short frames. The duration of each frame is usually 20-30 milliseconds. For audio with a sampling frequency of 16kHz, each frame contains 320-480 sampling points, so that a single audio signal generates several short frames, and generates a short frame sequence; S12, extracting features from each short frame in the short frame sequence, and confirming the audio comprehensive features of the audio signal based on the MFCC feature value associated with each short frame: S121, perform FFT on each short frame, convert the time domain signal into a frequency domain signal, and obtain a spectrum; the formula of FFT is: , where X[k] is the frequency domain signal, x[n] is the time domain signal, N is the length of the frame, n is the index of the time domain signal, indicating the time point, k is the index of the frequency domain signal, indicating the frequency point, and j is the imaginary unit, indicating the complex exponential function; S122. Then square the FFT result to get the power spectrum: P[k]=|X[k]| 2 ; S123. The power spectrum is passed through a set of Mel filters to convert the linear frequency into Mel frequency. The Mel frequency is more consistent with the perception characteristics of the human ear to sound. The conversion formula of Mel frequency is: , where f is the linear frequency, f mel It is the Mel frequency. Its Mel filter bank usually consists of 20-40 triangular filters, and the center frequency of each filter is evenly distributed according to the Mel frequency. S124, then take the logarithm of the output of each Mel filter to obtain the logarithmic energy: , where H m [k] is the frequency response of the mth Mel filter, and E[m] is the logarithmic energy; S125, performing discrete cosine transform on the logarithmic energy to obtain MFCC coefficients, wherein the formula of discrete cosine transform is: , where c[n] is the nth MFCC coefficient, M is the number of Mel filters, here we take the first 12-13 MFCC coefficients as features, because these coefficients contain the main spectral characteristics of the audio signal, the first 12-13 MFCC coefficients are averaged to confirm the MFCC feature value of this short frame; S126, using the same processing steps as steps S121-S125 for the several short frames in the short frame sequence to sequentially confirm the MFCC feature values ​​of the corresponding short frames, and then summing the MFCC feature values ​​of the several short frames to lock the audio comprehensive features of the audio signal; S13, based on the audio comprehensive features associated with the corresponding audio signal, check the audio signal with a preset feature check list to confirm the check characters associated with the audio signal, wherein the feature check list is a preset list prepared in advance by relevant operators based on experience, and sort the check characters associated with the continuous audio signals to generate the audio feature text of this stage, wherein the interval time between the continuous audio signals is less than 1 second; Specifically, by using the above-mentioned specific content and related features, the specific text associated with the corresponding voice content can be locked, and then the content can be indexed according to the corresponding text, and the content can be quickly displayed, so as to achieve a better content output effect; Step 2: perform character verification on the audio feature text and the cloud database, select the verification text from the cloud database, confirm the position of the verification text, and mark the confirmed position as the index position. Specifically, each different verification text has a different storage path node, and different storage path nodes are related to different storage locations. In the subsequent indexing process, such storage locations can be directly indexed. The specific sub-steps for marking the index position are: S21, mark the confirmed audio feature text as the main text, and confirm the text titles of other verification texts from the cloud database, use the confirmed text titles as the secondary text, and select similarity texts from the determined groups of secondary texts: perform character verification on the secondary text and the main text, mark the same characters as the same type of characters, and record the total number of the same type of characters G, then mark the total number of characters of the secondary text as H1, and the total number of characters of the main text as H2, use: ZB1=G÷H1 and ZB2=G÷H2 to confirm the proportion values ​​of the same type of characters ZB1 and ZB2, then lock the average proportion based on ZB1 and ZB2, and mark the secondary text with the average proportion ≥60% as the similarity text; S22, confirming the storage location of the similarity text from the cloud database, and marking the confirmed storage location as the index location; Specifically, each different text content has a different storage location, and different storage locations store specific content associated with the corresponding text. Therefore, the specific content associated with the corresponding storage location can be indexed, and the corresponding content can be indexed out to ensure a specific indexing rate; Step 3: Based on the confirmed index position and the preset index path, the text content associated with the index position is divided into types, multiple groups of type content are confirmed, and multiple groups of type content are combined according to the total number of index paths, the combined content is confirmed, and the multiple combined contents are integrated into a content group to be indexed, wherein the specific sub-steps of confirmation are: S31. Based on the text content associated with the index position, confirm the different types included in the corresponding text content (different data types correspond to different data formats), classify the different types of content, classify the same type of content into the same category, lock multiple groups of different types of type content, and set a division mark during the division process (to facilitate the subsequent combination of multiple groups of different types of type content according to the division mark); S32. Confirm the total number of preset index paths and mark it as Zs, then confirm the total number of different types of type content and mark it as ZG. If Zs≥ZG, there is no need to perform type combination, directly mark the corresponding type content as combined content, and integrate multiple combined contents into a content group to be indexed (that is, the total number of index paths exceeds the total number of type content. In this case, different types of type content can be directly indexed and output according to different index paths without type combination). If Zs<ZG, then multiple groups of different types of type content are randomly combined, and several combination processes are continued. In each combination process, the total number of multiple groups of combined content is always consistent with ZG, and the best process is selected from each combination process: S321, confirm the data capacity of different combination contents in each combination process, and mark the confirmed different data capacity as R i , where i represents different combinations of content, and then the confirmed multiple groups of data capacity R i Perform variance processing to lock the process variance of this combined process; S322, confirm the process variance associated with each different combination process in turn, and select the smallest process variance from the confirmed different process variances, and mark the combined process associated with the smallest process variance as the best process. Specifically, i=1, 2, ..., q is proposed, where q represents the total number of combined contents, and then the confirmed multiple groups of data capacities are averaged to confirm the average capacity Jz. Confirm the process variance associated with the corresponding multiple sets of data capacity; S33, marking the multiple groups of combined contents associated with the best process as content groups to be indexed; Step 4: According to the marked content group to be indexed, the combined content within the content group to be indexed is indexed and output according to the preset index path. Based on the index features generated during the indexing process, the combined content corresponding to the index path is determined to be most suitable for indexing and outputting, the determined combined content is marked as execution content, and indexed and outputted. The specific sub-steps of determining the execution content are as follows: S41. Based on the total number of index paths Zs, select part of the content from a single combined content, and divide the selected part of the content into Zs sub-contents, distribute the Zs sub-contents to the associated index paths in sequence, and output the content through the associated index paths, and record multiple groups of index rates of the corresponding index paths, average the multiple groups of index rates, determine the index average rate of the corresponding index path, and mark it as SY o , where o represents different index paths. For this combination, select SY o The index path associated with max is used as the execution path of this combined content, and this combined content is marked as the execution content, and the execution content is indexed and output according to this execution path; S42, when each group of execution paths exists in the index path, the total number thereof is reduced by 1, and the remaining combined contents are processed in the same manner as step S41, the execution paths associated with the corresponding combined contents and the determined execution contents are confirmed, and the determined execution contents are indexed and output according to the determined execution paths; According to this processing method, different combinations of content are processed in sequence, and content indexing is performed for rapid output to ensure output rate; Specifically, it is assumed that there are three groups of content in the proposed content group to be indexed, corresponding to three groups of preset index paths respectively. For a single group of content among the three groups of content, the single group of content is selected to complete the specific determination of part of the content, and then the determined part of the content is divided into three groups of index content, and then the divided different index contents are indexed and output according to different index paths. Each different index path will generate a different index rate. From the generated different index rates, the maximum value, that is, a group of index paths with the fastest index rate, is selected, and the corresponding combined content is indexed and output according to the selected index path to ensure the corresponding index rate; Subsequently, different combinations of content are associated with different index paths, so that each combination of content can reach the best indexing state, thereby achieving a faster indexing effect. The indexed related content will be directly displayed on the blackboard model for the corresponding teachers to select.

[0009] Some of the data in the above formulas are numerically calculated by removing their dimensions. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0010] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An audio and video online learning method based on blackboard model collaboration, characterized in that: The following steps are involved: Step 1: Acquire the audio signal generated during the class, and generate a specific audio feature text based on the relevant features of the acquired audio signal; Step 2: perform character verification on the audio feature text and the cloud database, select the verification text from the cloud database, confirm the position of the verification text, and mark the confirmed position as the index position; Step 3: Based on the confirmed index position and the preset index path, the text content associated with the index position is divided into types, multiple groups of type content are confirmed, and the multiple groups of type content are combined according to the total number of index paths, the combined content is confirmed, and the multiple combined content is integrated into a content group to be indexed; Step 4: According to the marked content group to be indexed, the combined content within the content group to be indexed is indexed and output according to a preset index path. Based on the index features generated during the indexing process, the combined content that best suits the corresponding index path for index output is determined, the determined combined content is marked as execution content, and indexed and output.

2. The method for online audio and video learning based on blackboard model collaboration according to claim 1 is characterized in that: In step 1, the specific sub-steps of generating specific audio feature text are: S11, collecting audio signals according to standard audio sampling parameters, with the sampling bit number being 16 bits, dividing the collected single audio signal into short frames, so that the single audio signal generates a plurality of short frames, and generating a short frame sequence; S12, extracting features from each short frame in the short frame sequence, and confirming the audio comprehensive features of the audio signal based on the MFCC feature value associated with each short frame; S13, based on the audio comprehensive features associated with the corresponding audio signal, checking the audio signal with a preset feature checklist to confirm the check character associated with the audio signal, The feature check table is a preset table, and the check characters associated with the continuous audio signals are sorted to generate audio feature text, wherein the interval time between the continuous audio signals is less than 1 second.

3. The method for online audio and video learning based on blackboard model collaboration according to claim 2 is characterized in that: In step S12, the audio comprehensive features of the audio signal are confirmed: S121, perform FFT on each short frame, convert the time domain signal into a frequency domain signal, and obtain a spectrum; the formula of FFT is: , where X[k] is the frequency domain signal, x[n] is the time domain signal, N is the length of the frame, n is the index of the time domain signal, indicating the time point, k is the index of the frequency domain signal, indicating the frequency point, and j is the imaginary unit, indicating the complex exponential function; S122. Then square the FFT result to get the power spectrum: P[k]=|X[k]| 2 ; S123, the power spectrum is passed through a set of Mel filters to convert the linear frequency into Mel frequency. The conversion formula of Mel frequency is: , where f is the linear frequency, f mel is the Mel frequency; S124, then take the logarithm of the output of each Mel filter to obtain the logarithmic energy: , where H m [k] is the frequency response of the mth Mel filter, and E[m] is the logarithmic energy; S125, performing discrete cosine transform on the logarithmic energy to obtain MFCC coefficients, wherein the formula of discrete cosine transform is: , where c[n] is the nth MFCC coefficient, M is the number of Mel filters, and the first 12-13 MFCC coefficients are averaged to determine the MFCC feature value of this short frame; S126, using the same processing steps as steps S121-S125 for the several short frames in the short frame sequence to sequentially confirm the MFCC feature values ​​of the corresponding short frames, and then summing the MFCC feature values ​​of the several short frames to lock the audio comprehensive features of the audio signal.

4. The method for online audio and video learning based on blackboard model collaboration according to claim 1 is characterized in that: In step 2, the specific sub-steps for calibrating the index position are: S21, mark the confirmed audio feature text as the main text, and confirm the text titles of other verification texts from the cloud database, use the confirmed text titles as the secondary text, and select similarity texts from the determined groups of secondary texts: perform character verification on the secondary text and the main text, mark the same characters as the same type of characters, and record the total number of the same type of characters G, then mark the total number of characters of the secondary text as H1, and the total number of characters of the main text as H2, use: ZB1=G÷H1 and ZB2=G÷H2 to confirm the proportion values ​​of the same type of characters ZB1 and ZB2, then lock the average proportion based on ZB1 and ZB2, and mark the secondary text with the average proportion ≥60% as the similarity text; S22. Confirm the storage location of the similarity text from the cloud database, and mark the confirmed storage location as the index location.

5. The method for online audio and video learning based on blackboard model collaboration according to claim 1 is characterized in that: In step 3, the specific method of confirming the combined content is: S31, based on the text content associated with the index position, confirm the different types included in the corresponding text content, divide the different types of content, divide the same type of content into the same category, lock multiple groups of different types of type content, and set division marks during the division process; S32. Confirm the total number of preset index paths and mark it as Zs, then confirm the total number of type contents of different types and mark it as ZG. If Zs≥ZG, there is no need to perform type combination, and directly mark the corresponding type content as combined content, and integrate multiple combined contents into a content group to be indexed.

6. The method for online audio and video learning based on blackboard model collaboration according to claim 5 is characterized in that: In the step S32, if Zs<ZG, multiple groups of different types of type content are randomly combined, and several combination processes are continued. In each combination process, the total number of multiple groups of combined content is always consistent with ZG, and the best process is selected from each combination process; S33: Marking multiple groups of combined contents associated with the best process as content groups to be indexed.

7. The method for online audio and video learning based on blackboard model collaboration according to claim 6 is characterized in that: In step S32, the specific method of selecting the best process is: S321, confirm the data capacity of different combination contents in each combination process, and mark the confirmed different data capacity as R i , where i represents different combinations of content, and then the confirmed multiple groups of data capacity R i Perform variance processing to lock the process variance of this combined process; S322. Confirm the process variances associated with each different combination of processes in turn, select the smallest process variance from the confirmed different process variances, and mark the combination process associated with the smallest process variance as the best process.

8. The method for online audio and video learning based on blackboard model collaboration according to claim 1, characterized in that: In step 4, the specific sub-steps for confirming the execution content are: S41. Based on the total number of index paths Zs, select part of the content from a single combined content, and divide the selected part of the content into Zs sub-contents, distribute the Zs sub-contents to the associated index paths in sequence, and output the content through the associated index paths, and record multiple groups of index rates of the corresponding index paths, average the multiple groups of index rates, determine the index average rate of the corresponding index path, and mark it as SY o , where o represents different index paths. For this combination, select SY o The index path associated with max is used as the execution path of this combined content, and this combined content is marked as the execution content, and the execution content is indexed and output according to this execution path; S42, when each group of execution paths exists in the index path, the total number thereof is reduced by 1, and the remaining combined contents are processed in the same manner as step S41, the execution paths associated with the corresponding combined contents and the determined execution contents are confirmed, and the determined execution contents are indexed and output according to the determined execution paths; According to this processing method, different combinations of content are processed in turn, and the content is indexed and quickly output. The indexed multiple groups of combined content are reorganized according to the set division marks, the text content is confirmed, and the confirmed text content is directly displayed on the blackboard model.

Images