Lughter detection method and device based on deep acoustic features, equipment and medium
By combining a deep acoustic feature-based laughter detection method with a feature extraction algorithm and a speech pre-training model, along with a deep residual model, the problem of low accuracy in laughter detection in noisy environments is solved, achieving higher accuracy in laughter detection.
Patent Information
- Application Number
- CN202411329257.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-07
AI Technical Summary
The accuracy of laughter detection is low in noisy environments. Background noise, echoes and reverberation interference are severe, and the characteristics of laughter in different emotional states are complex, which increases the difficulty of detection.
A laughter detection method based on deep acoustic features is adopted. Acoustic features are extracted through feature extraction algorithms and speech pre-training models, and laughter features are identified by combining deep residual models, thereby realizing multi-level feature analysis and fusion.
It improves the accuracy of laughter detection, enabling a more comprehensive understanding and recognition of laughter, and enhances detection capabilities in noisy environments.
Smart Images
Figure CN119274577A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing, and in particular to a laughter detection method, device, equipment and medium based on deep acoustic features. Background Art
[0002] Laughter detection is an important part of conversational speech synthesis systems. Laughter detection technology has important application value in many fields, such as sentiment analysis, user experience research, mental health monitoring, etc. Laughter, as an additional compensation for speech, can provide more emotional expression. For example, in the financial field, it can help the system automatically identify and analyze people's laughter during communication, thereby providing richer emotional information and helping financial staff understand the psychological state of the conversation partner. Therefore, research in different fields such as laughter recognition, measurement and segmentation plays an important role.
[0003] Laughter segmentation can provide me with laughter information related to the speaker. With the labeled laughter information, I can add additional laughter to conversational speech synthesis. I can insert additional laughter expressions of varying degrees into normal synthesized speech, making the expression of the conversational speech synthesis system more natural.
[0004] However, laughter detection usually needs to be carried out in a noisy environment. Background noise, echo and reverberation will interfere with the recognition of laughter. Moreover, laughter is not just a simple physiological reaction, it is also closely related to emotional state. Different emotional states may produce laughter with different characteristics, which increases the difficulty of detection.
[0005] Therefore, how to improve the accuracy of laughter detection has become a technical problem that needs to be solved urgently. Summary of the invention
[0006] The present application provides a laughter detection method, apparatus, device and medium based on deep acoustic features, aiming to improve the accuracy of laughter detection.
[0007] In a first aspect, the present application provides a laughter detection method based on deep acoustic features, and the laughter detection method based on deep acoustic features comprises the following steps:
[0008] Based on a feature extraction algorithm, extract features from at least one segmented acoustic signal to obtain a first acoustic feature;
[0009] Based on the speech pre-training model, deep acoustic feature extraction is performed on the segmented acoustic signal to obtain a second acoustic feature;
[0010] Based on the laughter detection model, laughter feature recognition is performed on the fusion feature of the first acoustic feature and the second acoustic feature to obtain laughter detection information.
[0011] In a second aspect, the present application further provides a laughter detection device based on deep acoustic features, the laughter detection device based on deep acoustic features comprising:
[0012] A first feature extraction module, configured to extract features from at least one segmented acoustic signal based on a feature extraction algorithm to obtain a first acoustic feature;
[0013] A second feature extraction module, used to perform deep acoustic feature extraction on the segmented acoustic signal based on a speech pre-training model to obtain a second acoustic feature;
[0014] The laughter detection module is used to perform laughter feature recognition on the fusion features of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain laughter detection information.
[0015] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the laughter detection method based on deep acoustic features as described above are implemented.
[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the above-mentioned laughter detection method based on deep acoustic features are implemented.
[0017] The present application provides a laughter detection method, device, computer equipment and storage medium based on deep acoustic features. The method of the present application includes extracting features from at least one segmented acoustic signal based on a feature extraction algorithm to obtain a first acoustic feature; extracting deep acoustic features from the segmented acoustic signal based on a speech pre-training model to obtain a second acoustic feature; and identifying laughter features based on a laughter detection model to obtain laughter detection information based on the fusion features of the first acoustic feature and the second acoustic feature. In the above manner, the present application uses a feature extraction algorithm to capture the first acoustic feature of an acoustic signal; extracts deep acoustic features from the signal through a speech pre-training model to mine deeper speech characteristics to obtain a second acoustic feature. By combining the feature extraction algorithm and the speech pre-training model, a multi-level feature analysis of the acoustic signal is achieved. The first acoustic feature and the second acoustic feature are fused and input into the laughter detection model for feature recognition, which integrates information at different levels so that the model can understand and recognize laughter more comprehensively, thereby improving the accuracy of laughter detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 A schematic flow chart of a first embodiment of a laughter detection method based on deep acoustic features provided by the present application;
[0020] Figure 2 A schematic diagram of a laughter detection process provided in an embodiment of the present application;
[0021] Figure 3 A schematic flow chart of a second embodiment of a laughter detection method based on deep acoustic features provided by the present application;
[0022] Figure 4 is a structural schematic diagram of a first embodiment of a laughter detection device based on deep acoustic features provided by the present application;
[0023] Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.
[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0028] Please refer to Figure 1 , Figure 1 This is a flow chart of a first embodiment of a laughter detection method based on deep acoustic features provided in the present application.
[0029] like Figure 1 As shown, the laughter detection method based on deep acoustic features includes steps S101 to S103.
[0030] S101, performing feature extraction on at least one segmented acoustic signal based on a feature extraction algorithm to obtain a first acoustic feature;
[0031] In one embodiment, feature extraction requires analyzing the speech signal, and firstly, feature parameters that can represent the essence of the speech need to be analyzed and extracted.
[0032] In one embodiment, original audio to be detected is collected; based on an audio sampling algorithm, the original audio is sampled to generate a discrete acoustic signal; based on a preset framing method, the discrete acoustic signal is framed to obtain at least one frame of the segmented acoustic signal.
[0033] In one embodiment, the user's original audio can be collected by a microphone or other sound receiving device, or the original audio file to be detected can be directly obtained, such as a call recording, etc. Then, the original audio is sampled by an audio sampling algorithm to generate a discrete audio signal.
[0034] In one embodiment, the acoustic signal contains too little available information and the time sequence of the sound signal is too long. For example, the sampling rate is 16000, which means that there are 16000 sampling points in a 1-second sound signal. Conventional models cannot process such a long sequence information. Therefore, it is necessary to perform frame processing on the discrete acoustic signal.
[0035] In one embodiment, the speech signal is a nonlinear, non-stationary and time-varying random signal, but can be regarded as stationary and time-invariant in a short time, so a stationary acoustic signal can be obtained through frame processing.
[0036] In one embodiment, since the speech signal has short-term stability, the speech is usually framed and windowed to capture short-term audio segments. Generally, the frame shift is 5-10ms, and the window length is 2-3 times the frame shift, i.e., 20-30ms.
[0037] It is understandable that even if the speech is quasi-periodic in a short period of time, it is difficult to ensure that each frame is completely periodic when dividing the speech into frames.
[0038] Generally, after the segmented acoustic signal is truncated, after the short-time Fourier transform STFT (Short Time Fourier Transform), there will be frequencies that are not in the original signal, that is, redundant frequency components. This phenomenon is called spectrum leakage. By adding a suitable window, these components, that is, high-order harmonics, can be suppressed to make the result smoother. When the segmented acoustic signal is subjected to a short-time Fourier transform, a series of discrete spectral lines are obtained. If the frequency components in the signal are located between the spectral lines instead of on the spectral lines, it will cause deviations in amplitude and aroma. This is the fence effect.
[0039] It is understandable that the errors caused by spectrum leakage and fence effect cannot be eliminated, but they can be suppressed by selecting a suitable window function. Therefore, after the discrete acoustic signal is framed, each segmented acoustic signal needs to be windowed.
[0040] In one embodiment, a window function (eg, a Hamming window) may be applied to each segmented acoustic signal to reduce discontinuities at frame boundaries.
[0041] Optionally, a suitable window function may be selected according to actual needs and applied to each segmented acoustic signal, such as a rectangular window, a Hamming window, and a Hanning window.
[0042] In one embodiment, the windowing operation is to multiply the segmented acoustic signal with the window function, thereby obtaining a windowed segmented acoustic signal.
[0043] Furthermore, based on Fourier transform, the segmented acoustic signal is spectrally transformed to obtain a segmented acoustic spectrum; based on Mel transform, the segmented acoustic spectrum is transformed to generate the first acoustic feature.
[0044] Generally speaking, the meaning of Fourier transform is that any signal can be decomposed into the superposition of several sine waves of different frequencies. In speech signals, the sine waves with the lowest frequency are called the fundamental wave of the signal, and the rest are called harmonics of the signal. There is only one fundamental wave, which can be called the first harmonic. There can be many harmonics, and the frequency of each harmonic is an integer multiple of the fundamental wave frequency. The size of the harmonics may be different. A series of bar graphs drawn with the frequency of the harmonic as the horizontal axis and the amplitude (size) as the vertical axis is called a spectrum. The spectrum can accurately reflect the internal structure of the signal.
[0045] In one embodiment, the discrete acoustic signal is divided into small blocks after windowing and framing, that is, short-period segmented acoustic signals. Each segmented acoustic signal is converted into a segmented acoustic spectrum through short-time Fourier transform, and the segmented acoustic spectrum is transformed into Mel spectrum information through Mel transform. Mel spectrum not only retains certain timing information of the sound signal, but also represents more frequency information, so acoustic features are usually used instead of the original acoustic signal as the input of the model.
[0046] In one embodiment, in the field of speech signal processing, it is usually necessary to convert the signal into a corresponding spectrum form to obtain a spectrogram, and use the data on the spectrogram as the signal feature. The horizontal axis x of the spectrogram is time, the vertical axis y is frequency, and the value corresponding to (x, y) represents the amplitude of frequency y at time x.
[0047] In one embodiment, the short-time Fourier transform (STFT) divides a long signal into frames, adds windows, performs a Fourier transform (FFT) on each frame, and finally stacks the results of each frame along another dimension to obtain a two-dimensional signal form similar to a picture, namely a spectrogram.
[0048] In one embodiment, the spectrogram is a large image. In order to obtain sound features of appropriate size, it is usually necessary to perform a Mel transform on the spectrogram, convert the ordinary frequency scale into a Mel frequency scale, and transform the spectrogram into a Mel spectrum.
[0049] Among them, the essence of short-time Fourier transform (STFT) is the windowed Fourier transform, which performs Fourier transform in different windows in the form of sliding windows. The specific process is: before the signal is Fourier transformed, it is multiplied by a time-limited window function, and it is assumed that the non-stationary signal is stationary within the short time interval of the analysis window. Through the movement of the window function on the time axis, the signal is analyzed segment by segment to obtain a set of local spectra of the discrete acoustic signal.
[0050] S102, performing deep acoustic feature extraction on the segmented acoustic signal based on the speech pre-training model to obtain a second acoustic feature;
[0051] In one embodiment, the speech pre-training model may be a speech recognition model such as wav2vec2.0.
[0052] In one embodiment, wav2vec2.0 is an unsupervised speech pre-training model that can be migrated to downstream tasks of speech. wav2vec2.0 passes the continuous speech signal through a quantization module to achieve the conversion of the feature space from infinite continuous to finite discrete, fixes the pre-training network parameters, and then connects several convolution layers to extract deep acoustic features in the speech signal.
[0053] In one embodiment, wav2vec 2.0 is a self-supervised audio pre-training model that learns deep representations of audio by training on large-scale unlabeled audio data.
[0054] Generally speaking, the architecture of the model mainly consists of three parts: feature extractor, quantizer and context network.
[0055] The feature extractor is responsible for converting the original audio signal into a feature representation suitable for model processing. In wav2vec 2.0, the feature extractor uses the structure of a convolutional neural network (CNN) to extract local features of the audio by performing convolution operations on the audio signal. The quantizer converts the continuous feature representation output by the feature extractor into a discrete latent representation. This discretization process enables the model to learn the intrinsic structure of the audio signal, thereby achieving effective encoding of the audio signal. The context network is responsible for capturing the contextual information of the audio signal. By calculating the self-attention mechanism on the discrete latent representation, the context network can learn the global structure and semantic content of the audio signal.
[0056] In one embodiment, the second acoustic feature may include, but is not limited to, a temporal feature, a local feature, a discrete latent representation, and context information.
[0057] S103: Based on a laughter detection model, perform laughter feature recognition on a fusion feature of the first acoustic feature and the second acoustic feature to obtain laughter detection information.
[0058] It is understandable that after the number of layers of ordinary neural network models is stacked to a certain extent, due to the effect of the activation function, the gradient will disappear. For this reason, the embodiment of the present application selects a deep residual model as a laughter detection model to use deeper network layers to better learn laughter detection; at the same time, the increase in model depth also means that the scale of the model is expanded, which can effectively increase the recognition accuracy of the model.
[0059] In one embodiment, the laughter detection model detects speech input frames. When laughter is detected in several consecutive frames, not only the laughter information in the speech can be detected, but also the start and end positions of the laughter can be marked from the starting speech frame and the ending speech frame, thus completing the laughter detection and labeling functions.
[0060] Furthermore, based on the time correspondence, the first acoustic feature and the second acoustic feature corresponding to the same segmented acoustic signal are feature fused to obtain the fused features corresponding to each segmented acoustic signal; based on the laughter detection model, the fused features are feature classified to obtain the laughter detection information.
[0061] In one embodiment, if Figure 2 As shown, when extracting the first acoustic feature and the second acoustic feature of the segmented acoustic signal, the first acoustic feature and the second acoustic feature corresponding to the same segmented acoustic signal are fused according to the time correspondence, and then the fused feature is input into the laughter detection model (i.e., the deep residual model) for classification detection to determine whether there is laughter feature in the input segmented acoustic signal, that is, to detect whether there is laughter in the segmented acoustic signal.
[0062] In one embodiment, the first acoustic feature and the second acoustic feature corresponding to the same segmented acoustic signal can be merged into a set of fused features through simple data concatenation, weighted averaging or more complex fusion strategies (such as principal component analysis PCA, fusion layers in deep learning, etc.).
[0063] In another embodiment, a laughter detection model may also be used to perform laughter detection on the first acoustic feature and the second acoustic feature of the same segmented acoustic signal respectively, and then the detection results are combined through voting, decision trees or other logical rules to output final laughter detection information.
[0064] In one embodiment, the laughter detection information may include presence (whether laughter exists in the segmented acoustic signal), location information, intensity, duration, frequency information, laughter type, context information, etc.
[0065] Exemplarily, the location information may include a timestamp and a time period, i.e., the specific time point at which the laughter occurs, and the timestamps of the beginning and end of the laughter, which are used to represent the starting point and the end point of the time period. The type of laughter may be used to distinguish different types of laughter, such as chuckles, laughter, etc. The context information may include the identity information of the laughter speaker, and may also include the emotional state, such as happiness, irony, etc.
[0066] Furthermore, based on the laughter detection model, a fusion feature of the first acoustic feature and the second acoustic feature is vector-converted to obtain a fusion feature vector; when the eigenvalue of the fusion feature vector is greater than or equal to a preset laughter feature threshold, the fusion feature is determined to be a laughter feature; based on the segmented acoustic signal corresponding to the laughter feature, the laughter detection information is obtained.
[0067] In one embodiment, the first acoustic feature and the second acoustic feature can be combined according to the time correspondence through simple concatenation, weighted average or other complex fusion techniques to form a fusion feature. The fusion feature is normalized to make its value range uniform for model processing. The laughter detection model is used to encode the fusion feature and convert it into a fixed-length feature vector.
[0068] In one embodiment, a preset laughter feature threshold may be set to determine whether the fused feature vector represents laughter. The laughter feature threshold may be set based on experience or may be optimized through methods such as cross-validation.
[0069] In one embodiment, the eigenvalue in the fused feature vector is evaluated to determine whether it is greater than or equal to a preset laughter feature threshold. If the eigenvalue of the fused feature vector is greater than or equal to the laughter feature threshold, the fused feature is determined to be a laughter feature, and a laughter signal exists in the corresponding segmented acoustic signal. On the contrary, if the eigenvalue of the fused feature vector is less than the laughter feature threshold, the fused feature is determined not to be a laughter feature, and a laughter signal does not exist in the corresponding segmented acoustic signal.
[0070] In one embodiment, the laughter detection information includes laughter location information.
[0071] In one embodiment, the segmented acoustic signal corresponding to the laughter feature is marked as a laughter acoustic signal; when it is detected that the fusion feature corresponding to multiple consecutive segmented acoustic signals is a laughter feature, the starting segmented acoustic signal and the ending segmented acoustic signal corresponding to the laughter feature are marked; based on the segment sequence numbers corresponding to the starting segmented acoustic signal and the ending segmented acoustic signal, the laughter position information is obtained.
[0072] In one embodiment, based on the detected laughter features, the corresponding segmented acoustic signal is determined, which may include the timestamp and duration of the laughter, and laughter detection information is extracted from the segmented acoustic signal, which may include but is not limited to the laughter type (such as chuckle, laugh, etc.), laughter intensity, and frequency characteristics.
[0073] In one embodiment, each segmented acoustic signal is analyzed using a laughter detection model to determine whether it contains a laughter feature. If a laughter feature is detected, the corresponding segmented acoustic signal is marked as a laughter acoustic signal.
[0074] In one embodiment, when a laughter acoustic signal is detected, the corresponding segmented acoustic signal can be marked, and then laughter detection is performed on the next segmented acoustic signal. If there is no laughter signal in the next segmented acoustic signal, the marking is ended, and only the laughter signal is marked.
[0075] If there is still a laughter signal in the next segmented acoustic signal, the next segmented acoustic signal is continuously marked, and the first segmented acoustic signal in which a laughter signal is detected is marked as the starting segmented acoustic signal; then laughter detection is continued for subsequent segmented acoustic signals until it is detected that there is no laughter signal in a certain segmented acoustic signal, and the previous segmented acoustic signal of the segmented acoustic signal is marked as the ending segmented acoustic signal. In this way, a continuous segmented acoustic signal can be obtained, in which there is a continuous laughter signal.
[0076] In another embodiment, all segmented acoustic signals may be traversed to check the continuous laughter acoustic signal, and the start and end positions of the continuous laughter acoustic signal may be recorded.
[0077] In one embodiment, the sequence numbers of the start and end segmented acoustic signals can be used to calculate the time period in which the laughter occurs. If the segmented acoustic signals correspond to time stamps, these time stamps can be directly used to determine the specific location of the laughter.
[0078] Exemplarily, the laughter location information may be stored in a suitable data structure, such as a list, a database or a file. The laughter location information may be output as required, such as displayed on a user interface or generated as a report.
[0079] This embodiment provides a laughter detection method based on deep acoustic features. The method of this application uses a feature extraction algorithm to capture the first acoustic feature of an acoustic signal; a speech pre-training model is used to extract deep acoustic features of the signal, and deeper speech characteristics are mined to obtain a second acoustic feature. By combining the feature extraction algorithm and the speech pre-training model, a multi-level feature analysis of the acoustic signal is achieved. The first acoustic feature and the second acoustic feature are fused and input into the laughter detection model for feature recognition, which integrates information at different levels so that the model can understand and recognize laughter more comprehensively, thereby improving the accuracy of laughter detection.
[0080] Please refer to Figure 3 , Figure 3 This is a flow chart of a second embodiment of a laughter detection method based on deep acoustic features provided in the present application.
[0081] In this embodiment, Figure 3 As shown, based on the above Figure 1 In the illustrated embodiment, before S103, the following steps are further included:
[0082] S201, collecting laughter audio and non-laughter audio;
[0083] In one embodiment, a large amount of audio data is collected and annotated to determine which audio data is laughter audio and which data is non-laughter audio, as a model training data set.
[0084] In one embodiment, the laughter audio samples in the model training data set may have different characteristics such as laughter types, background noise, speakers, etc.
[0085] In one embodiment, before training the model using a model training data set, the laughter audio and non-laughter audio may be preprocessed using a data preprocessing method, including denoising, volume normalization, segmentation, etc., to improve the quality and consistency of feature extraction.
[0086] S202, based on an audio processing algorithm, extracting features of the laughter audio and the non-laughter audio respectively to obtain laughter features and non-laughter features;
[0087] In one embodiment, the audio processing algorithm may include algorithms such as MFCC (Mel Frequency Cepstral Coefficient), Spectrogram (Short Time Fourier Transform Function), Chroma Feature (Chroma Feature), etc.
[0088] In one embodiment, the audio data may be enhanced, such as adding noise, changing pitch, speed, etc., to increase the generalization ability of the model.
[0089] S203. Based on the laughter features and the non-laughter features, iteratively optimize and train the pre-trained classification model until the classification accuracy of the pre-trained classification model for the laughter features reaches a preset accuracy threshold, and use the pre-trained classification model as the laughter detection model.
[0090] In one embodiment, a residual model structure may be designed, including multiple residual blocks, each of which includes several convolutional layers and skip connections. Batch normalization and activation functions may be added between residual blocks. Thus, a deep residual model is constructed as a pre-trained classification model.
[0091] In one embodiment, the residual model is trained using the extracted laughter features and non-laughter features. The data set is divided into a training set, a validation set, and a test set. A suitable loss function (such as cross entropy loss) and an optimizer (such as Adam or SGD) are selected for model training. During the training process, the model is evaluated using the validation set, and the model parameters are adjusted and optimized according to the classification accuracy on the validation set. Techniques such as learning rate decay, early stopping, etc. can be used to prevent overfitting. The model is continuously trained and optimized until a preset accuracy threshold is reached on the validation set.
[0092] In one embodiment, during the actual application of the laughter detection model, the model capability can be evaluated in combination with user feedback, and then the model can be continuously optimized and adjusted based on the audio data detected by the model and the user's feedback information, so that the laughter detection model always maintains high-accuracy laughter detection performance.
[0093] See also Figure 4 , Figure 4 It is a structural schematic diagram of a first embodiment of a laughter detection device based on deep acoustic features provided in the present application. The laughter detection device based on deep acoustic features is used to execute the aforementioned laughter detection method based on deep acoustic features.
[0094] like Figure 4 As shown, the laughter detection device 300 based on deep acoustic features includes: a first feature extraction module 301, a second feature extraction module 302 and a laughter detection module 303.
[0095] A first feature extraction module 301, configured to extract features from at least one segmented acoustic signal based on a feature extraction algorithm to obtain a first acoustic feature;
[0096] A second feature extraction module 302, configured to perform deep acoustic feature extraction on the segmented acoustic signal based on a speech pre-training model to obtain a second acoustic feature;
[0097] The laughter detection module 303 is used to perform laughter feature recognition on the fusion feature of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain laughter detection information.
[0098] In one embodiment, the first feature extraction module 301 includes:
[0099] A spectrum conversion unit, used for performing spectrum conversion on the segmented acoustic signal based on Fourier transform to obtain a segmented acoustic spectrum;
[0100] A Mel transform unit is used to transform the segmented acoustic spectrum based on Mel transform to generate the first acoustic feature.
[0101] In one embodiment, the laughter detection device 300 based on deep acoustic features further includes an audio framing module, including:
[0102] A first audio collection unit, used for collecting original audio to be detected;
[0103] An audio sampling unit, used for sampling the original audio based on an audio sampling algorithm to generate a discrete acoustic signal;
[0104] The audio framing unit is used to perform framing processing on the discrete acoustic signal based on a preset framing method to obtain at least one frame of the segmented acoustic signal.
[0105] In one embodiment, the laughter detection module 303 includes:
[0106] A feature fusion unit, configured to fuse the first acoustic feature and the second acoustic feature corresponding to the same segmented acoustic signal based on a time correspondence, to obtain the fused feature corresponding to each segmented acoustic signal;
[0107] The first laughter detection unit is used to perform feature classification on the fusion feature based on the laughter detection model to obtain the laughter detection information.
[0108] In one embodiment, the laughter detection module 303 further includes:
[0109] A feature vectorization unit, configured to perform vector conversion on a fusion feature of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain a fusion feature vector;
[0110] A second laughter detection unit, configured to determine that the fused feature is a laughter feature when the feature value of the fused feature vector is greater than or equal to a preset laughter feature threshold;
[0111] The laughter detection information obtaining unit is used to obtain the laughter detection information based on the segmented acoustic signal corresponding to the laughter feature.
[0112] In one embodiment, the laughter detection information includes laughter location information;
[0113] The laughter detection information obtaining unit comprises:
[0114] A signal marking subunit, used for marking the segmented acoustic signal corresponding to the laughter feature as a laughter acoustic signal;
[0115] A laughter signal recognition subunit, for marking a starting segmented acoustic signal and an ending segmented acoustic signal corresponding to the laughter feature when detecting that the fusion feature corresponding to a plurality of consecutive segmented acoustic signals is a laughter feature;
[0116] The laughter position information obtaining subunit is used to obtain the laughter position information based on the segment sequence numbers corresponding to the starting segment acoustic signal and the ending segment acoustic signal.
[0117] In one embodiment, the laughter detection device 300 based on deep acoustic features further includes a model training module, including:
[0118] A second audio collection unit, used for collecting laughter audio and non-laughter audio;
[0119] A third feature extraction unit is used to extract features of the laughter audio and the non-laughter audio respectively based on an audio processing algorithm to obtain laughter features and non-laughter features;
[0120] The model training unit is used to iteratively optimize the pre-trained classification model based on the laughter features and the non-laughter features until the classification accuracy of the pre-trained classification model for the laughter features reaches a preset accuracy threshold, and the pre-trained classification model is used as the laughter detection model.
[0121] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the laughter detection method based on deep acoustic features, and will not be repeated here.
[0122] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 5 Runs on the computer device shown.
[0123] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0124] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0125] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any laughter detection method based on deep acoustic features.
[0126] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0127] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any laughter detection method based on deep acoustic features.
[0128] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0129] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0130] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0131] Based on a feature extraction algorithm, extract features from at least one segmented acoustic signal to obtain a first acoustic feature;
[0132] Based on the speech pre-training model, deep acoustic feature extraction is performed on the segmented acoustic signal to obtain a second acoustic feature;
[0133] Based on the laughter detection model, laughter feature recognition is performed on the fusion feature of the first acoustic feature and the second acoustic feature to obtain laughter detection information.
[0134] In one embodiment, when the processor implements the feature extraction algorithm to extract features from at least one segmented acoustic signal to obtain the first acoustic feature, the processor is used to implement:
[0135] Based on Fourier transform, the segmented acoustic signal is subjected to spectrum transformation to obtain a segmented acoustic spectrum;
[0136] The segmented acoustic spectrum is transformed based on Mel transform to generate the first acoustic feature.
[0137] In one embodiment, before implementing the feature extraction algorithm to extract features from at least one segmented acoustic signal to obtain the first acoustic feature, the processor is further configured to implement:
[0138] Collect the original audio to be detected;
[0139] Based on the audio sampling algorithm, the original audio is sampled to generate a discrete acoustic signal;
[0140] Based on a preset framing method, the discrete acoustic signal is subjected to framing processing to obtain at least one frame of the segmented acoustic signal.
[0141] In one embodiment, when the processor performs laughter feature recognition on the fusion feature of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain laughter detection information, it is used to implement:
[0142] Based on the time correspondence, the first acoustic feature and the second acoustic feature corresponding to the same segmented acoustic signal are subjected to feature fusion to obtain the fusion feature corresponding to each segmented acoustic signal;
[0143] Based on the laughter detection model, the fused features are classified to obtain the laughter detection information.
[0144] In one embodiment, when the processor implements the laughter detection model based on which the fusion feature of the first acoustic feature and the second acoustic feature is used to perform laughter feature recognition to obtain laughter detection information, the processor is further used to implement:
[0145] Based on the laughter detection model, performing vector conversion on a fusion feature of the first acoustic feature and the second acoustic feature to obtain a fusion feature vector;
[0146] When the feature value of the fused feature vector is greater than or equal to a preset laughter feature threshold, determining that the fused feature is a laughter feature;
[0147] The laughter detection information is obtained based on the segmented acoustic signal corresponding to the laughter feature.
[0148] In one embodiment, the laughter detection information includes laughter location information;
[0149] When the processor implements the acquisition of the laughter detection information based on the segmented acoustic signal corresponding to the laughter feature, the processor is used to implement:
[0150] Marking the segmented acoustic signal corresponding to the laughter feature as a laughter acoustic signal;
[0151] When it is detected that the fusion feature corresponding to a plurality of consecutive segmented acoustic signals is a laughter feature, marking a starting segmented acoustic signal and an ending segmented acoustic signal corresponding to the laughter feature;
[0152] The laughter position information is obtained based on the segment sequence numbers corresponding to the start segment acoustic signal and the end segment acoustic signal.
[0153] In one embodiment, before implementing the laughter detection model based on which the fusion feature of the first acoustic feature and the second acoustic feature is used to perform laughter feature recognition to obtain laughter detection information, the processor is further used to implement:
[0154] Collect laughter audio and non-laughter audio;
[0155] Based on an audio processing algorithm, feature extraction is performed on the laughter audio and the non-laughter audio respectively to obtain laughter features and non-laughter features;
[0156] Based on the laughter features and the non-laughter features, the pre-trained classification model is iteratively optimized and trained until the classification accuracy of the pre-trained classification model for the laughter features reaches a preset accuracy threshold, and the pre-trained classification model is used as the laughter detection model.
[0157] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the laughter detection methods based on deep acoustic features provided in the embodiments of the present application.
[0158] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A laughter detection method based on deep acoustic features, characterized in that: The method comprises: Based on a feature extraction algorithm, extract features from at least one segmented acoustic signal to obtain a first acoustic feature; Based on the speech pre-training model, deep acoustic feature extraction is performed on the segmented acoustic signal to obtain a second acoustic feature; Based on the laughter detection model, laughter feature recognition is performed on the fusion feature of the first acoustic feature and the second acoustic feature to obtain laughter detection information.
2. The laughter detection method based on deep acoustic features according to claim 1, characterized in that: The step of extracting features from at least one segmented acoustic signal based on a feature extraction algorithm to obtain a first acoustic feature includes: Based on Fourier transform, the segmented acoustic signal is subjected to spectrum transformation to obtain a segmented acoustic spectrum; The segmented acoustic spectrum is transformed based on Mel transform to generate the first acoustic feature.
3. The laughter detection method based on deep acoustic features according to claim 1, characterized in that: The method further comprises: extracting features from at least one segmented acoustic signal based on a feature extraction algorithm to obtain a first acoustic feature. Collect the original audio to be detected; Based on the audio sampling algorithm, the original audio is sampled to generate a discrete acoustic signal; Based on a preset framing method, the discrete acoustic signal is subjected to framing processing to obtain at least one frame of the segmented acoustic signal.
4. The laughter detection method based on deep acoustic features according to claim 1, characterized in that: The step of performing laughter feature recognition on the fusion feature of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain laughter detection information includes: Based on the time correspondence, the first acoustic feature and the second acoustic feature corresponding to the same segmented acoustic signal are subjected to feature fusion to obtain the fusion feature corresponding to each segmented acoustic signal; Based on the laughter detection model, the fused features are classified to obtain the laughter detection information.
5. The laughter detection method based on deep acoustic features according to claim 1, characterized in that: The method further comprises: performing laughter feature recognition on the fusion feature of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain laughter detection information; Based on the laughter detection model, performing vector conversion on a fusion feature of the first acoustic feature and the second acoustic feature to obtain a fusion feature vector; When the feature value of the fused feature vector is greater than or equal to a preset laughter feature threshold, determining that the fused feature is a laughter feature; The laughter detection information is obtained based on the segmented acoustic signal corresponding to the laughter feature.
6. The laughter detection method based on deep acoustic features according to claim 5, characterized in that: The laughter detection information includes laughter location information; The step of obtaining the laughter detection information based on the segmented acoustic signal corresponding to the laughter feature comprises: Marking the segmented acoustic signal corresponding to the laughter feature as a laughter acoustic signal; When it is detected that the fusion feature corresponding to a plurality of consecutive segmented acoustic signals is a laughter feature, marking a starting segmented acoustic signal and an ending segmented acoustic signal corresponding to the laughter feature; The laughter position information is obtained based on the segment sequence numbers corresponding to the start segment acoustic signal and the end segment acoustic signal.
7. The laughter detection method based on deep acoustic features according to claim 1, characterized in that: Before performing laughter feature recognition on the fusion feature of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain laughter detection information, the method further includes: Collect laughter audio and non-laughter audio; Based on an audio processing algorithm, feature extraction is performed on the laughter audio and the non-laughter audio respectively to obtain laughter features and non-laughter features; Based on the laughter features and the non-laughter features, the pre-trained classification model is iteratively optimized and trained until the classification accuracy of the pre-trained classification model for the laughter features reaches a preset accuracy threshold, and the pre-trained classification model is used as the laughter detection model.
8. A laughter detection device based on deep acoustic features, characterized in that: The laughter detection device based on deep acoustic features comprises: A first feature extraction module, configured to extract features from at least one segmented acoustic signal based on a feature extraction algorithm to obtain a first acoustic feature; A second feature extraction module, used for performing deep acoustic feature extraction on the segmented acoustic signal based on a speech pre-training model to obtain a second acoustic feature; The laughter detection module is used to perform laughter feature recognition on the fusion features of the first acoustic feature and the second acoustic feature based on the laughter detection model to obtain laughter detection information.
9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the laughter detection method based on deep acoustic features according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the laughter detection method based on deep acoustic features as claimed in any one of claims 1 to 7 are implemented.