Large model high-quality reasoning method and system supporting edge computing devices

Abstract

The present application relates to the field of large model data processing technology, and specifically to a large model high-quality reasoning method and system supporting edge computing devices, the method comprising: obtaining a speech acquisition vector; obtaining the time-frequency feature value of each element at each frequency at the corresponding moment according to the frequency domain information stability and self-similar characteristics of all elements in time windows of different lengths centered on each element in the speech acquisition vector; obtaining the emotional significance at each frequency according to the distribution discreteness of the time-frequency feature value and energy amplitude of all elements at each frequency at the corresponding moment; filtering the energy amplitude of each frequency in the speech spectrum of the speech acquisition vector according to the emotional significance at each frequency and then inversely transforming to obtain the filtered speech acquisition vector; and using edge computing devices to perform large model reasoning on the filtered speech acquisition vector. The present application can improve the accuracy of large model reasoning calculations.

Description

Technical Field

[0001] The present application relates to the technical field of large model data processing, and specifically to a large model high-quality reasoning method and system that supports edge computing devices. Background Art

[0002] High-quality reasoning of large models refers to extracting features from a large amount of data and training a large model to predict or classify new data. In actual classroom education, a large amount of voice data is usually generated by multiple classes at the same time during class time. If the voice data is uniformly transmitted to the upper end for large model processing, it will occupy a large amount of communication, storage, and computing resources. Therefore, edge computing is required for classroom voice data. When performing edge computing processing, since the performance of edge computing devices is usually lower than that of upper end devices, knowledge distillation of the upper end large model is required.

[0003] The purpose of knowledge distillation is to transfer the knowledge of the large model to the small model, using lower computing resources to ensure a certain data processing capability. During the distillation process, the small model not only learns the training data in the large model, but also learns the label information and output information of the large model training process through knowledge transfer.

[0004] With the popularization and development of multimedia education, the voice data generated in the classroom will be subject to various noise interferences. The training data formed by the high-interference classroom voice data has a great impact on the data quality in the large model training set, making the data features, label information and output information learned by the small model have lower confidence, resulting in inaccurate reasoning calculations of the large model. Summary of the invention

[0005] In order to solve the above technical problems, the purpose of this application is to provide a large model high-quality reasoning method and system that supports edge computing devices. The technical solutions adopted are as follows:

[0006] In a first aspect, an embodiment of the present application provides a large model high-quality reasoning method supporting edge computing devices, the method comprising the following steps:

[0007] S1, obtain the speech collection vector;

[0008] S2, according to the stability of the frequency domain information of all elements in the time windows of different lengths centered on each element in the speech acquisition vector, the phoneme prominence of each element at each frequency at the corresponding moment is obtained; according to the comparison of the self-similar features of all elements in the time windows of different lengths centered on each element, the speech time domain features of each element at the corresponding moment are obtained;

[0009] S3, comprehensively analyzing the speech time domain features of each element at the corresponding moment and its phoneme prominence at each frequency to obtain the time-frequency feature value of each element at the corresponding moment at each frequency; according to the distribution discreteness of the time-frequency feature value and energy amplitude of all elements at each frequency at the corresponding moment, obtain the emotion prominence at each frequency;

[0010] S4, obtaining the speech spectrum of the speech collection vector; filtering the energy amplitude of each frequency in the speech spectrum of the speech collection vector according to the emotion significance at each frequency and then inverse transforming it to obtain the filtered speech collection vector; using the edge computing device to perform large model inference on the filtered speech collection vector.

[0011] Furthermore, the method for obtaining the phoneme prominence of each element at each frequency at the corresponding moment includes:

[0012] Recording a vector composed of all elements contained in a short-time window preset with each element as the center as the short-time window vector of each element; extracting frequency domain information from the short-time window vector of each element to obtain a speech spectrum of the short-time window vector of each element;

[0013] Obtain the standard deviation of the energy amplitude corresponding to each frequency in the speech spectrum of the short-time window vector of all elements in the short-time window and the long-time window preset with each element as the center, as the short-time frequency stability and long-time frequency stability of each element at each frequency at the corresponding moment; wherein the length of the long-time window is greater than the length of the short-time window;

[0014] According to the comparison between the long-term frequency stability and the short-term frequency stability at each frequency at the corresponding moment of each element, the phoneme significance at each frequency at the corresponding moment of each element is obtained.

[0015] Further, the method of obtaining the phoneme prominence of each element at each frequency at the corresponding moment according to the comparison between the long-term frequency stability and the short-term frequency stability of each element at each frequency at the corresponding moment includes: recording the phoneme prominence of the nth element in the speech collection vector at the kth frequency at the corresponding moment as X n,k , Among them, A n,k , B n,k They are respectively the long-term frequency stability and short-term frequency stability at the k-th frequency at the moment corresponding to the n-th element in the speech acquisition vector; μ is a constant preset to be greater than 0.

[0016] Furthermore, the method for calculating the speech time domain features of each element at a corresponding moment includes:

[0017] A fractal window of a first preset length is constructed with each element in the speech acquisition vector as the center; the length of the fractal window is greater than the length of the short-time window; the short-time window of each element and the vector composed of all elements in the fractal window are respectively calculated using the box counting method to obtain the short-time window of each element and the fractal dimension of the fractal window; the ratio of the fractal dimension of the fractal window of each element to the fractal dimension of the short-time window is used as the speech time domain feature of each element at the corresponding moment.

[0018] Further, the time-frequency feature value of each element at each frequency at the corresponding moment is obtained by integrating the speech time-domain features of each element at the corresponding moment and its phoneme prominence at each frequency, including: taking the product of the speech time-domain features of each element at the corresponding moment and its phoneme prominence at each frequency as the time-frequency feature value of each element at each frequency at the corresponding moment.

[0019] Furthermore, the method for obtaining the emotional significance at each frequency includes:

[0020] Normalize the time-frequency feature value of each element in the speech acquisition vector at each frequency at the corresponding time to obtain the forward feature weight of each element at each frequency;

[0021] The product of the energy amplitude at each frequency in the speech spectrum of the short-time window vector of each element in the speech acquisition vector and the forward feature weight of each element at each frequency is taken as the weighted energy value of each element at each frequency; the standard deviation of the weighted energy values ​​of all elements at each frequency is calculated as the emotional significance at each frequency.

[0022] Furthermore, the method for obtaining the forward feature weight of each element at each frequency includes:

[0023] The sum of the time-frequency eigenvalues ​​of each element at all frequencies is calculated; and the ratio of the time-frequency eigenvalue at each frequency to the sum is used as the forward feature weight of each element at each frequency at the corresponding moment.

[0024] Further, the energy amplitude of each frequency in the speech spectrum of the speech collection vector is filtered and then inversely transformed according to the emotion significance at each frequency to obtain the filtered speech collection vector, including:

[0025] The emotion saliency at all frequencies is normalized by negative mapping to obtain the energy filtering weight at each frequency; the product of the energy amplitude of each frequency in the speech spectrum of the speech acquisition vector and the energy filtering weight is used as the filtering energy of each frequency; the filtering energies of all frequencies are combined into a filtered speech spectrum; the filtered speech spectrum is inversely transformed to obtain a filtered speech acquisition vector.

[0026] Furthermore, the negative mapping normalization is performed on the emotion saliency at all frequencies to obtain the energy filtering weight at each frequency, including:

[0027] The maximum value of the emotion significance at all frequencies is obtained; the difference between the maximum value and the emotion significance at each frequency is calculated; the sum of the differences at all frequencies is recorded as the first sum; the ratio of the emotion significance at each frequency to the first sum is used as the energy filtering weight at each frequency.

[0028] In the second aspect, an embodiment of the present application also provides a large-model high-quality reasoning system that supports edge computing devices, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, it implements the steps of any one of the above-mentioned large-model high-quality reasoning methods that support edge computing devices.

[0029] This application has at least the following beneficial effects:

[0030] The present application aims to solve the problem of inaccurate large model inference calculation caused by low quality of classroom speech data. By analyzing the characteristics of non-human voice factors in the speech acquisition vector as noise that interferes with emotion recognition in the frequency domain, the data in the short-time window of each element in the speech acquisition vector is processed to obtain the spectrum of the speech classroom vector at different times. The phoneme significance is calculated by the energy stability duration of the spectrum to characterize the human voice components contained in the speech data at different frequency positions at the same time. Further, a short-time window and a fractal window are set in the time domain based on the duration of the phoneme of the human voice. The fractal dimension of the signal in the fractal window is used to reflect the low self-similarity of the speech signal in the long-time time domain. The short-time window is used to represent the low self-similarity of the speech signal in the long-time time domain. The signal fractal dimension of the time window reflects the high self-similarity of the speech signal in a shorter time domain window. The two fractal dimensions are further fused to obtain the time domain characteristics of the speech. The time domain characteristics of the speech and the phoneme salience are combined to obtain the time-frequency feature values, which characterize the content of speech components in different frequency positions at different times. Finally, in view of the characteristic that the energy fluctuations of the speech signal represent emotions, the emotion salience is obtained by combining the time-frequency feature values ​​to process the speech acquisition vector, and the signal energy components representing the human voice are filtered and enhanced to complete the data feature enhancement operation related to emotions in the speech acquisition vector, thereby improving the quality of speech data and avoiding the problem of inaccurate reasoning and calculation of large models due to low quality of speech data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A flowchart of a method for high-quality reasoning of large models supporting edge computing devices provided in one embodiment of the present application;

[0033] Figure 2 A schematic diagram for comparing the fractal dimensions of a short-time window and a fractal window provided for an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0036] The specific scheme of the large-model high-quality reasoning method and system supporting edge computing devices provided by the present application is described in detail below with reference to the accompanying drawings.

[0037] See also Figure 1 , which shows a flowchart of a method for supporting high-quality reasoning of a large model of an edge computing device provided by an embodiment of the present application, the method comprising the following steps:

[0038] S1, obtain the speech collection vector.

[0039] This application aims to filter high-interference speech data and generate diversified digital labels to form high-quality training samples, and train a large model with excellent performance based on high-quality training samples; secondly, optimize the large model by knowledge distillation, and compress and distill the large model to achieve rapid deployment of the large model on edge computing devices with limited computing resources and computing power. The filtering processing method of this application is applicable to the field of artificial intelligence, and to the recognition of human voices in public places such as airports, train stations, classrooms, and autonomous driving, so as to improve the quality of speech data used to train the model during the large model reasoning process.

[0040] As an embodiment of the present application, classroom voice data in the classroom is taken as an example for filtering processing. Specifically, classroom voice data is obtained by using multimedia equipment, and the emotional characteristics and audio characteristics in the classroom voice data are analyzed to achieve filtering processing of classroom voice data. A large model is trained based on a large amount of filtered audio data, which is recorded as the upper-end model. Then, a small model is deployed in the edge computing device to migrate the knowledge in the upper-end model to the small model.

[0041] Therefore, when processing voice data, this embodiment obtains classroom voice data through multimedia equipment and stores it in the form of a vector. The collection time is 40 minutes, the sampling frequency is 48 kHz, and the voice collection vector is obtained. The vector length is N=115200, and the nth element represents the sound field intensity at the nth moment.

[0042] S2, according to the stability of the frequency domain information of all elements in time windows of different lengths centered on each element in the speech acquisition vector, obtain the phoneme significance of each element at each frequency at the corresponding moment; according to the comparison of the self-similar features of all elements in time windows of different lengths centered on each element, obtain the speech time domain features of each element at the corresponding moment.

[0043] Since the performance of the small model deployed in the edge computing device depends on the performance of the upper-end model, the high-interference, low-signal-to-noise ratio classroom speech data contains a large amount of interference information, which can easily cause the large model to learn inaccurate data features during the training and inference process, resulting in poor performance of the small model and low data processing capabilities after being deployed on the edge device.

[0044] Therefore, this embodiment preprocesses the speech acquisition vector, improves the signal-to-noise ratio of classroom speech data, enhances the speech features related to emotional information in classroom speech data, and thus improves the reasoning accuracy of small models in edge computing devices. Non-human voice factors in speech acquisition vectors are noise that interferes with emotion recognition, so it is necessary to filter non-human voice factors in speech acquisition vectors.

[0045] Language in audio is usually composed of multiple phoneme delays. Phonemes are usually reflected in the spectrum as sound components with a maximum continuous time of 50 milliseconds and stable frequency. Non-human voice noise usually lasts for a long time. Therefore, this embodiment first extracts frequency domain features from classroom speech data.

[0046] Specifically, a short-time window of a preset length is constructed with each element in the speech collection vector as the center; in this embodiment, the time length of the short-time window is 50 milliseconds, that is, each short-time window contains W=2401 elements. In particular, when the number of elements in the window is less than 2401, the elements in the window are supplemented by the mean filling method.

[0047] The vector composed of all elements contained in the short-time window centered on each element is recorded as the short-time window vector of each element; the frequency domain information of the short-time window vector of each element is extracted to obtain the speech spectrum of the short-time window vector of each element; taking the nth short-time window vector as an example, the FFT fast Fourier transform algorithm is used to extract the frequency domain information, the necessary parameter frequency range is 0 to 20k Hz, the frequency sampling interval is 10 Hz, and the speech spectrum of the nth short-time window vector is obtained. The FFT algorithm is a commonly used technology for speech processing. The data form of the speech spectrum is a vector with a length of K=2000, and its kth element represents the energy amplitude of the short-time window vector at the kth frequency.

[0048] Further, the standard deviation of the energy amplitude corresponding to each frequency in the speech spectrum of the short-time window vector of all elements in the short-time window and the long-time window preset with each element as the center is obtained respectively as the short-time frequency stability and long-time frequency stability of each element at each frequency at the corresponding moment; wherein the length of the long-time window is greater than the length of the short-time window. In this embodiment, the length of the short-time window is 50 milliseconds, and the length of the long-time window is 100 milliseconds.

[0049] It can be understood that the smaller the short-term frequency stability, the more stable the characteristics of the speech collection vector at a certain frequency position in the neighborhood of each element corresponding to the time, the more consistent with the frequency characteristics of the phoneme, and the more speech phoneme components are contained in this frequency position. The greater the long-term frequency stability, the more unstable the characteristics of the speech collection vector at a certain frequency position in the large neighborhood of each element corresponding to the time, the more consistent with the frequency characteristics of the phoneme, and the more speech phoneme components are contained in this frequency position.

[0050] Finally, the phoneme significance at the kth frequency corresponding to the nth element in the speech acquisition vector is recorded as X n,k , Among them, A n,k , B n,k They are respectively the long-term frequency stability and the short-term frequency stability at the k-th frequency at the moment corresponding to the n-th element in the speech acquisition vector; μ is a preset constant greater than 0, and the value of this embodiment is 0.1.

[0051] Among them, short-time frequency stability represents the frequency stability of speech in a short-time window, and long-time frequency stability represents the frequency stability of speech in a long-time window. The more stable the short-time frequency in the short-time window is and the more unstable the short-time frequency in the long-time window is, the more the signal of speech at a certain frequency position conforms to the characteristic that the phoneme maintains frequency stability with a short-time window as the time length. This means that the more speech phoneme components each element contains at the frequency position at the corresponding moment, the greater the phoneme significance is.

[0052] Speech is characterized in the form of phonemes in the frequency domain, and in the time domain, due to the low repetitiveness of speech itself, the self-similarity of the signal is low. Therefore, the human voice content of speech at different times can be identified through the self-similarity of speech signals in the time domain, and the time domain characteristics of speech can be obtained. Further combining the time domain and frequency domain characteristics of speech can reflect the time-frequency characteristics of human voice in the speech acquisition vector.

[0053] First, in the time domain, human speech usually uses phonemes as the smallest repeating unit. When the length of the speech signal is less than the smallest repeating unit, the self-similarity of the signal is usually high, otherwise the self-similarity is low.

[0054] In order to reflect the characteristic of low self-similarity of speech itself, the signal window taken should be longer than the minimum phoneme of speech. Therefore, a fractal window of a first preset length is constructed with each element in the speech acquisition vector as the center; the length of the fractal window is greater than the length of the short-time window; the length of the fractal window in this embodiment is T3 = 500 milliseconds. That is, it contains 24001 elements. In particular, when the number of elements in the window is less than 24001, the elements in the window are supplemented by the mean filling method. As two-dimensional data, the speech signal can be characterized by self-similarity by calculating the fractal dimension; the larger the fractal dimension of a two-dimensional data, the lower its self-similarity.

[0055] The fractal dimension of the short-time window of each element and the vector composed of all elements in the fractal window are further calculated using the box counting method to obtain the fractal dimension of the short-time window and fractal window of each element. The box counting method is a commonly used technology in the field of speech processing, and the specific implementation process will not be repeated. The ratio of the fractal dimension of the fractal window of each element to the fractal dimension of the short-time window is used as the speech time domain feature of each element at the corresponding moment. Figure 2 As shown in the figure, the audio signal contains speech, select the fractal window in the dotted frame and the short-time window in the solid frame set at one moment, and calculate the fractal dimension. Obviously, the fractal dimension of the fractal window vector is larger, and this position contains more human voice components.

[0056] Among them, the larger the fractal dimension of the fractal window, the lower the self-similarity of the speech collection vector, and the more human voice components it contains; the smaller the fractal dimension of the short-time window, the higher the self-similarity of the speech collection vector in a short time, and the more likely the small self-similarity feature reflected in the fractal window is caused by the syllable combination of speech rather than noise, and the larger the obtained speech time domain feature, the more speech human voice components the speech collection vector contains.

[0057] S3, comprehensively analyzing the speech time domain features of each element at the corresponding moment and its phoneme prominence at each frequency to obtain the time-frequency feature value of each element at the corresponding moment at each frequency; according to the distribution discreteness of the time-frequency feature values ​​and energy amplitudes of all elements at each frequency at the corresponding moment, obtain the emotion prominence at each frequency.

[0058] In order to extract classroom emotions at different times from the speech collection vector, the short-term energy of the sound signal is an important feature to characterize emotions. Generally, the greater the energy, the more excited the speaker is; conversely, the more frustrated. Therefore, this embodiment extracts the features of the speech collection vector from the energy perspective, which is conducive to performing large model inference calculations through edge devices to obtain the analysis results of the speech collection vector.

[0059] The energy of human voice usually fluctuates at intervals of phonemes. Therefore, this embodiment extracts the energy fluctuations of the phoneme part in the voice collection vector as the energy fluctuation characteristics of human voice, and completes the emotional feature extraction of the voice collection vector.

[0060] Specifically, the product of the speech time domain feature of each element at the corresponding moment and its phoneme significance at each frequency is taken as the time-frequency feature value of each element at the corresponding moment at each frequency; the time-frequency feature value of each element at the corresponding moment in the speech acquisition vector at each frequency is normalized to obtain the positive feature weight of each element at each frequency.

[0061] As an embodiment of the present application, the method for normalizing the time-frequency feature values ​​of each element at all frequencies at the corresponding moment is: calculating the sum of the time-frequency feature values ​​of each element at all frequencies; and taking the ratio of the time-frequency feature value at each frequency to the sum as the positive feature weight of each element at each frequency at the corresponding moment.

[0062] As other embodiments of the present application, a linear normalization function may be selected, and the present application does not make any limitation thereto.

[0063] The frequency position with a large positive feature weight is closer to the frequency of the phoneme, and the energy amplitude change can better represent the energy change characteristics of the emotional characteristics.

[0064] Therefore, this embodiment further uses the product of the energy amplitude at each frequency in the speech spectrum of the short-time window vector of each element in the speech acquisition vector and the forward feature weight of each element at each frequency as the weighted energy value of each element at each frequency. The standard deviation of the weighted energy values ​​of all elements at each frequency is calculated as the emotional significance at each frequency.

[0065] The standard deviation of the energy sequence is weighted with the positive feature weight as the weight. The larger the product of the standard deviation of the energy sequence and the positive feature weight, the more the energy amplitude of the frequency position in the speech collection vector reflects the energy amplitude of the emotional characteristics. The greater the emotional significance obtained at the end, the more the energy amplitude of the frequency position can reflect the emotional characteristics in the speech collection vector.

[0066] S4, obtaining the speech spectrum of the speech collection vector; filtering the energy amplitude of each frequency in the speech spectrum of the speech collection vector according to the emotion significance at each frequency and then inverse transforming it to obtain the filtered speech collection vector; using the edge computing device to perform large model inference on the filtered speech collection vector.

[0067] Finally, the emotion saliency is obtained at all frequency positions, and the energy amplitude at each frequency position reflects the emotion characteristics of the speech acquisition vector at different frequencies. Therefore, this embodiment filters the energy amplitude of each frequency in the speech spectrum of the speech acquisition vector by emotion saliency.

[0068] Specifically, the emotional saliency at all frequencies is normalized by negative mapping to obtain the energy filtering weight at each frequency; the method for normalizing the emotional saliency at all frequencies by negative mapping is as follows: obtaining the maximum value of the emotional saliency at all frequencies; calculating the difference between the maximum value and the emotional saliency at each frequency; recording the sum of the differences at all frequencies as the first sum; and taking the ratio of the emotional saliency at each frequency to the first sum as the energy filtering weight at each frequency. The larger the energy filtering weight, the less the energy amplitude change at the frequency position can reflect the emotion, and the more the energy amplitude change at the frequency position should be suppressed.

[0069] Finally, the speech acquisition vector is used as input and the Fast Fourier Transform (FFT) algorithm is used to calculate the speech spectrum of the speech acquisition vector, where the frequency range is 0 to 20k Hz and the frequency sampling interval is 10 Hz. The FFT algorithm is a common technology for speech processing. The data form of the speech spectrum is a vector with a length of K=2000.

[0070] Further, the product of the energy amplitude of each frequency in the speech spectrum of the speech acquisition vector and the energy filtering weight is used as the filtering energy of each frequency; the filtering energies of all frequencies are combined into a filtered speech spectrum;

[0071] Finally, the inverse transform algorithm of FFT is used to calculate the filtered speech spectrum to obtain the filtered speech acquisition vector; the inverse transform algorithm of FFT is a commonly used technology in the field of speech processing.

[0072] This embodiment uses knowledge distillation to obtain the distillation results of the large model, obtains a small model, and deploys it to the edge computing device. Knowledge distillation is a well-known technology. The specific method is as follows:

[0073] The speech collection vector is collected by multimedia equipment, and converted into a filtered speech collection vector according to the method described in this embodiment as the input sequence of the training data set; the collected speech collection vector is labeled, and the labeling method is: generate a vector of length N, and the element values ​​in the vector are integers from 1 to 9, corresponding to: hesitation, anger, silence, tension, doubt, satisfaction, surprise, neutrality, excitement. The element values ​​in the vector are manually labeled by technicians to form a sequence recorded as a label sequence;

[0074] Finally, this embodiment collects a total of 4000 speech collection vectors, corresponding to 4000 input sequences and label sequences, and encodes each input sequence and its corresponding label sequence. The encoding result is a high-quality training sample with interference removed and high signal-to-noise ratio. All high-quality training samples constitute the training data set.

[0075] Furthermore, in combination with the upper-end model and the training data set, the small model actually deployed in the edge computing device is obtained by knowledge distillation, wherein the upper-end model and the small model use the same network structure. In this embodiment, the network structure is an LSTM model, and the specific acquisition method is:

[0076] The LSTM model is trained with the training data set as input, and the output data of the model is recorded as the emotion classification value. Optional: The activation function uses the ReLU function, and the optimizer uses the Adam optimizer; the training of the neural network is a well-known technology, and the specific process is not repeated here.

[0077] During the distillation process, the loss function is composed of the addition of two parts. The first part is the sentence-level loss, and the second part is the audio frame-level loss. The sentence-level loss function is the mean square error function, and the calculation process is: through temporal pooling, the output of the large model and the speech embedding of the small model are obtained to obtain their respective average values, and then the mean square error of the two is calculated as the sentence-level loss. The audio frame-level loss is obtained by calculating the mean square error of the output of the large model and the output of the small model on the frame.

[0078] Furthermore, after the training is completed, a small model is obtained after distillation of the upper-end model. At this time, the small model has performance close to that of the large model, and the number of parameters is significantly less than that of the large model. The small model is deployed to the edge computing device, allowing the edge computing device with limited computing power to efficiently process audio data.

[0079] Since this embodiment performs feature extraction processing on the speech acquisition vector and feature enhancement processing on the energy change characteristics of the human voice component in the speech signal, it is possible to complete high-quality reasoning of classroom emotions in the edge device with a small amount of calculation. High-quality reasoning of classroom emotions is only one embodiment of this application. The method described in this application is also applicable to the emotional feature extraction of speech data in other scenarios in the field of artificial intelligence, which improves the data processing effect of the edge computing device and avoids the problem of inaccurate emotional reasoning of speech recognition data.

[0080] Based on the same inventive concept as the above method, an embodiment of the present application also provides a large model high-quality reasoning system supporting edge computing devices, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the large model high-quality reasoning methods supporting edge computing devices.

[0081] Through the above description of the implementation method in combination with the accompanying drawings, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0082] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A high-quality reasoning method for large models that supports edge computing devices, characterized by: The method comprises the following steps: S1, obtain the speech collection vector; S2, record the vector composed of all elements contained in the short-time window preset with each element as the short-time window vector of each element; extract the frequency domain information of the short-time window vector of each element to obtain the speech spectrum of the short-time window vector of each element; obtain the standard deviation of the energy amplitude corresponding to each frequency in the speech spectrum of the short-time window vector of all elements in the short-time window and long-time window preset with each element as the short-time frequency stability and long-time frequency stability of each element at each frequency at the corresponding moment; wherein the length of the long-time window is greater than the length of the short-time window; according to the comparison of the long-time frequency stability and the short-time frequency stability at each frequency at the corresponding moment of each element, obtain the phoneme significance of each element at each frequency at the corresponding moment; according to the comparison of the self-similar features of all elements in time windows of different lengths centered on each element, obtain the speech time domain features of each element at the corresponding moment; S3, comprehensively analyzing the speech time domain features of each element at the corresponding moment and its phoneme prominence at each frequency to obtain the time-frequency feature value of each element at the corresponding moment at each frequency; according to the distribution discreteness of the time-frequency feature value and energy amplitude of all elements at each frequency at the corresponding moment, obtain the emotion prominence at each frequency; S4, obtaining the speech spectrum of the speech collection vector; filtering the energy amplitude of each frequency in the speech spectrum of the speech collection vector according to the emotion significance at each frequency and then inverse transforming it to obtain the filtered speech collection vector; using the edge computing device to perform large model inference on the filtered speech collection vector.

2. The large-model high-quality reasoning method supporting edge computing devices as claimed in claim 1, characterized in that: The method of obtaining the phoneme significance of each element at each frequency at the corresponding moment according to the comparison of the long-term frequency stability and the short-term frequency stability of each element at each frequency at the corresponding moment includes: recording the phoneme significance of the nth element at the kth frequency at the corresponding moment in the speech collection vector as X n,k , Among them, A n,k , B n,k They are respectively the long-term frequency stability and short-term frequency stability at the k-th frequency at the moment corresponding to the n-th element in the speech acquisition vector; μ is a constant preset to be greater than 0.

3. The large-model high-quality reasoning method supporting edge computing devices as claimed in claim 1, characterized in that: The method for calculating the speech time domain features of each element at a corresponding moment includes: A fractal window of a first preset length is constructed with each element in the speech acquisition vector as the center; the length of the fractal window is greater than the length of the short-time window; the short-time window of each element and the vector composed of all elements in the fractal window are respectively calculated using the box counting method to obtain the short-time window of each element and the fractal dimension of the fractal window; the ratio of the fractal dimension of the fractal window of each element to the fractal dimension of the short-time window is used as the speech time domain feature of each element at the corresponding moment.

4. The large-model high-quality reasoning method supporting edge computing devices as claimed in claim 1, characterized in that: The method of synthesizing the speech time domain features of each element at the corresponding moment and its phoneme prominence at each frequency to obtain the time-frequency feature value of each element at the corresponding moment at each frequency includes: taking the product of the speech time domain features of each element at the corresponding moment and its phoneme prominence at each frequency as the time-frequency feature value of each element at the corresponding moment at each frequency.

5. The large-model high-quality reasoning method supporting edge computing devices as claimed in claim 1, characterized in that: The method for obtaining the emotion significance at each frequency includes: Normalize the time-frequency feature value of each element in the speech acquisition vector at each frequency at the corresponding time to obtain the forward feature weight of each element at each frequency; The product of the energy amplitude at each frequency in the speech spectrum of the short-time window vector of each element in the speech acquisition vector and the forward feature weight of each element at each frequency is taken as the weighted energy value of each element at each frequency; the standard deviation of the weighted energy values ​​of all elements at each frequency is calculated as the emotional significance at each frequency.

6. The large-model high-quality reasoning method supporting edge computing devices as claimed in claim 5, characterized in that: The method for obtaining the forward feature weight of each element at each frequency includes: The sum of the time-frequency eigenvalues ​​of each element at all frequencies is calculated; and the ratio of the time-frequency eigenvalue at each frequency to the sum is used as the forward feature weight of each element at each frequency at the corresponding moment.

7. The large-model high-quality reasoning method supporting edge computing devices as claimed in claim 1, characterized in that: The step of filtering the energy amplitude of each frequency in the speech spectrum of the speech collection vector according to the emotion significance at each frequency and then inversely transforming it to obtain the filtered speech collection vector comprises: The emotion saliency at all frequencies is normalized by negative mapping to obtain the energy filtering weight at each frequency; the product of the energy amplitude of each frequency in the speech spectrum of the speech acquisition vector and the energy filtering weight is used as the filtering energy of each frequency; the filtering energies of all frequencies are combined into a filtered speech spectrum; the filtered speech spectrum is inversely transformed to obtain a filtered speech acquisition vector.

8. The large-model high-quality reasoning method supporting edge computing devices as claimed in claim 7, characterized in that: The negative mapping normalization is performed on the emotional significance at all frequencies to obtain the energy filtering weight at each frequency, including: The maximum value of the emotion significance at all frequencies is obtained; the difference between the maximum value and the emotion significance at each frequency is calculated; the sum of the differences at all frequencies is recorded as the first sum; the ratio of the emotion significance at each frequency to the first sum is used as the energy filtering weight at each frequency.

9. A large-model high-quality reasoning system supporting edge computing devices, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the large-model high-quality reasoning method supporting edge computing devices are implemented as described in any one of claims 1-8.

Images