Audio highlight determination method, device, electronic device and storage medium

Abstract

The present invention discloses a method, device, electronic device and storage medium for determining audio highlights. The method comprises obtaining original audio data; slicing the original audio data to obtain sliced ​​audio, wherein the sliced ​​audio is in seconds; extracting features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features; inputting the original audio data and the Mel spectrum features into a preset deep model for audio recognition to determine the audio tags corresponding to each second of audio data in the original audio data; performing highlight point generation processing on the audio tags corresponding to each second of audio data in the original audio data to obtain audio highlight point segments. The solution provided by the present invention can be combined with the needs of business scenarios to complete automatic detection and recognition of audio highlights.

Description

Technical Field

[0001] The present invention relates to the technical field of audio detection, and in particular to a method, device, electronic device and storage medium for determining audio highlights. Background Art

[0002] Audio tagging is an important direction in audio pattern recognition. Its fundamental task is to analyze audio signals and identify the meaning of audio types, such as human voice, singing, music, etc. Existing audio tagging technologies are mainly based on signal processing methods, extracting audio features such as loudness and frequency, and analyzing audio types according to rules. Alternatively, the model can be trained using a data set, and the convolutional neural network in the visual depth model can be used to perform audio tagging tasks to improve the audio tagging effect.

[0003] However, in the prior art, the academic community has not conducted any relevant research on the detection of wonderful sounds (such as laughter, applause, and cheers). Summary of the invention

[0004] In order to solve the technical problem that the prior art cannot automatically detect and identify audio highlights, the embodiments of the present invention provide a method, device, electronic device and storage medium for determining audio highlights.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a method for determining audio highlights, the method comprising:

[0007] Get the original audio data;

[0008] Slicing the original audio data to obtain sliced ​​audio, where the sliced ​​audio is in seconds;

[0009] Extracting features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features;

[0010] Input the original audio data and the Mel spectrum feature into a preset deep model for audio recognition, and determine the audio label corresponding to the audio data per second in the original audio data;

[0011] Perform highlight point generation processing on the audio tags corresponding to the audio data per second in the original audio data to obtain audio highlight point segments.

[0012] In the above solution, the slicing of the original audio data to obtain the sliced ​​audio includes:

[0013] Get the set sliding window step size and window size;

[0014] According to the sliding window step size and the window size, the original audio data is divided into a plurality of one-dimensional arrays;

[0015] Obtain the number of the one-dimensional arrays and the number of seconds of the original audio data;

[0016] According to the number of the plurality of one-dimensional arrays and the number of seconds of the original audio data, a preset number of first arrays and / or second arrays are copied at the head and the tail of the plurality of one-dimensional arrays, so that the number of the one-dimensional arrays is the same as the number of seconds of the original audio data; wherein the first array is the first one-dimensional array obtained when the original audio data is segmented; and the second array is the last one-dimensional array obtained when the original audio data is segmented;

[0017] The multiple one-dimensional arrays copied head to tail are used as the sliced ​​audios of the original audio data.

[0018] In the above solution, dividing the original audio data into a plurality of one-dimensional arrays according to the sliding window step size and the window size includes:

[0019] Slide the original audio data according to the sliding window step size, and each time the sliding is performed, split the original audio data after sliding according to the window size to obtain a one-dimensional array;

[0020] The original audio data is slid multiple times according to the sliding window step size to obtain multiple one-dimensional arrays.

[0021] In the above solution, the step of extracting features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features includes:

[0022] Performing pre-emphasis processing on the sliced ​​audio to obtain first audio data;

[0023] Performing frame division and windowing operations on the first audio data to obtain second audio data;

[0024] Performing discrete Fourier transform on the second audio data to obtain frequency domain data;

[0025] The frequency domain data is identified using a Mel filter to obtain Mel spectrum features.

[0026] In the above scheme, the inputting of the original audio data and the Mel spectrum feature into a preset deep model for audio recognition, and determining the audio label corresponding to the audio data per second in the original audio data includes:

[0027] Using a shallow feature extraction module in the deep model to extract features from the original audio data, to obtain a first feature;

[0028] Performing data enhancement on the Mel frequency spectrum feature using the deep model to obtain a second feature;

[0029] The first feature and the second feature are subjected to feature processing in the deep model to determine an audio label corresponding to each second of audio data in the original audio data.

[0030] In the above scheme, the performing feature processing on the first feature and the second feature in the deep model to determine the audio label corresponding to the audio data per second in the original audio data includes:

[0031] Connecting the first feature and the second feature respectively to obtain a three-dimensional feature map;

[0032] Using a one-dimensional convolutional layer to extract features from the three-dimensional feature map in the time domain dimension to obtain three-dimensional time domain features in the time domain dimension;

[0033] Using a two-dimensional convolutional layer to extract features of the three-dimensional time domain features in the time domain and space dimensions, to obtain three-dimensional time domain and space features in the time domain and space dimensions;

[0034] Performing feature redundancy reduction and feature size unification processing on the three-dimensional time domain and space features to obtain uniform size features;

[0035] The uniform size features are spatially mapped and label identified to obtain an audio label corresponding to each second of audio data in the original audio data.

[0036] In the above scheme, the spatial mapping and label discrimination of the uniform size features to obtain the audio label corresponding to the audio data per second in the original audio data includes:

[0037] Performing spatial mapping on the uniform size feature to obtain a label value corresponding to the uniform size feature;

[0038] Get the set recall threshold;

[0039] Compare the label value with the recall threshold to determine the label binary value corresponding to the label value;

[0040] It is determined according to the binary value of the label whether the audio data per second corresponding to the label value has the audio label corresponding to the binary value of the label.

[0041] In the above scheme, the step of performing a highlight generation process on the audio tag corresponding to each second of the audio data in the original audio data to obtain an audio highlight segment includes:

[0042] Get the smoothing step size and smoothing window size;

[0043] Slide the original audio data according to the smoothing step size, and perform numerical unification processing on the binary values ​​of the original audio data of the smoothing window size each time the data is slid, to obtain the binary values ​​after the numerical unification;

[0044] Slide the original audio data multiple times according to the smoothing step length to obtain a binary value after the audio data value per second in the original audio data is unified;

[0045] According to the continuity of the binary values ​​of the audio data values ​​per second in the original audio data after being unified, the audio segment corresponding to the binary values ​​of the continuous preset values ​​is determined as the audio highlight segment.

[0046] The embodiment of the present invention further provides an audio highlight point determination device, the audio highlight point determination device comprising:

[0047] An acquisition module, used to acquire original audio data;

[0048] A slicing module, used for slicing the original audio data to obtain sliced ​​audio, where the sliced ​​audio is in seconds;

[0049] A feature extraction module, used to extract features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features;

[0050] An audio recognition module, used to input the original audio data and the Mel spectrum features into a preset deep model for audio recognition, and determine the audio label corresponding to the audio data per second in the original audio data;

[0051] The highlight generation module is used to perform highlight generation processing on the audio tags corresponding to the audio data per second in the original audio data to obtain audio highlight segments.

[0052] An embodiment of the present invention further provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein:

[0053] When the processor is used to run the computer program, the steps of any of the above methods are executed.

[0054] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0055] The audio highlight determination method, device, electronic device and storage medium provided in the embodiments of the present invention obtain original audio data; slice the original audio data to obtain sliced ​​audio, wherein the sliced ​​audio is in seconds; extract features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features; input the original audio data and the Mel spectrum features into a preset deep model for audio recognition to determine the audio tags corresponding to the audio data per second in the original audio data; perform highlight generation processing on the audio tags corresponding to the audio data per second in the original audio data to obtain audio highlight segments. The solution provided by the present invention can be combined with the needs of business scenarios to complete automatic detection and identification of audio highlights. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a flow chart of a method for determining audio highlights according to an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the operation steps of the system in the application embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the Mel spectrum extraction process in an application example of the present invention;

[0059] Figure 4 This is a schematic diagram of the preprocessing module flow in an application example of the present invention;

[0060] Figure 5 This is a schematic diagram of the deep model structure of an application embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of the flow of a second-level tag recognition module according to an application example of the present invention;

[0062] Figure 7 This is a schematic diagram of the post-processing module flow chart of an application example of the present invention;

[0063] Figure 8 This is a schematic diagram of the system framework of an application embodiment of the present invention;

[0064] Fig. 9 A schematic diagram of the structure of an audio highlight point determination device according to an embodiment of the present invention;

[0065] Fig.10 The figure is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0067] The embodiment of the present invention provides a method for determining audio highlights. Figure 1As shown, the method includes:

[0068] Step 101: Obtain original audio data;

[0069] Step 102: Slice the original audio data to obtain sliced ​​audio, where the sliced ​​audio is in seconds;

[0070] Step 103: extracting features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features;

[0071] Step 104: inputting the original audio data and the Mel-spectrogram feature into a preset deep model for audio recognition, and determining an audio label corresponding to each second of audio data in the original audio data;

[0072] Step 105: Perform highlight generation processing on the audio tags corresponding to the audio data per second in the original audio data to obtain audio highlight segment.

[0073] Specifically, when obtaining the original audio data, the audio data can be obtained directly, or the audio and video data can be obtained first, and then the audio data in the audio and video data can be separated by software to obtain the original audio data. Here, ffmpeg open source software can be used to separate the original audio data from the audio and video data.

[0074] In practical application, the original audio data can be encoded into a specific format for storage, and then the original audio data in the specific format can be audio-recognized to identify the highlights in the original audio data. Here, the highlights in this application refer to audio points with highlights such as laughter, applause, and cheers. In practical application, the original audio data can be encoded into a wav format for storage, and then the original audio data in the wav format can be audio-recognized to identify the highlights in the original audio data. When specifically stored, the original audio data can be stored in a one-dimensional array using an open source audio processing library and stored in memory.

[0075] After obtaining the original audio data, the original audio data may be sliced ​​to obtain sliced ​​audio, where the sliced ​​audio is in seconds.

[0076] Specifically, in order to achieve a higher reliability in the analysis of short-time audio and to realize the analysis and processing of second-level audio, the original audio data may be sliced ​​by seconds.

[0077] Further, in one embodiment, slicing the original audio data to obtain second-level audio data includes:

[0078] Get the set sliding window step size and window size;

[0079] According to the sliding window step size and the window size, the original audio data is divided into a plurality of one-dimensional arrays;

[0080] Obtain the number of the one-dimensional arrays and the number of seconds of the original audio data;

[0081] According to the number of the plurality of one-dimensional arrays and the number of seconds of the original audio data, a preset number of first arrays and / or second arrays are copied at the head and the tail of the plurality of one-dimensional arrays, so that the number of the one-dimensional arrays is the same as the number of seconds of the original audio data; wherein the first array is the first one-dimensional array obtained when the original audio data is segmented; and the second array is the last one-dimensional array obtained when the original audio data is segmented;

[0082] The multiple one-dimensional arrays copied head to tail are used as the sliced ​​audios of the original audio data.

[0083] In actual application, slicing can be performed based on business scenarios. Specifically, the sliding window step size can be set based on the detection granularity requirements of the business scenario; the window size can be set based on the depth model, the detection confidence in the business scenario, and / or the characteristics of the sample length; the original audio data is divided into one-dimensional audio data using the set sliding window step size and window size to obtain multiple one-dimensional arrays.

[0084] According to the sliding window step and the window size, when the original audio data is segmented, the original audio data can be slid according to the sliding window step, and each time the sliding is performed, the original audio data after the sliding is segmented according to the window size to obtain a one-dimensional array; the original audio data is slid multiple times according to the sliding window step to obtain multiple one-dimensional arrays. For example, based on the detection granularity requirements of the business scenario, the sliding window step can be set to 1 second; then based on the characteristics of the depth model, the detection confidence in the business scenario and / or the length of the sample, the window size is set to 2 seconds. Then, when the original audio is 10 seconds, the original audio data is slid at a speed of sliding 1 second of the original audio data each time, and each time the sliding is performed, a 2-second audio data is sliced ​​out from the original audio data after the sliding, and each sliced ​​2-second audio data is a one-dimensional array. Slicing 10 seconds of audio data according to the sliding window step of 1 second and the window size of 2 seconds can obtain a total of 9 one-dimensional arrays.

[0085] Since the present application performs audio recognition on the original audio data in seconds, in order to ensure that there is a corresponding recognition result for each second of the audio data in the original audio data, and to avoid the situation where the audio data of some seconds has no corresponding recognition result due to the reduction of the number of one-dimensional arrays caused by slicing processing, the present application can copy a preset number of first arrays and / or second arrays at the head and tail of multiple one-dimensional arrays, so that the number of the one-dimensional arrays is the same as the number of seconds of the original audio data, thereby achieving a corresponding recognition result for each second of the audio data. For example, when the sliding window step size is 1 second and the window size is 2 seconds, 10 seconds of original audio data is sliced ​​to obtain 9 one-dimensional arrays, and the first one-dimensional array can be copied at the head of the 9 one-dimensional arrays, or the last one-dimensional array can be copied at the tail of the 9 one-dimensional arrays, so that the number of the one-dimensional arrays is the same as the number of seconds of the original audio.

[0086] Here, when selecting the sliding window step and window size, the appropriate sliding window step and window size can be selected based on the application scenario. For example, for the second-level labeling scenario, when selecting the sliding window step and window size, the sliding window step can be set to be smaller than the window size, so that the one-dimensional arrays after the slice processing overlap with each other, thereby adapting to the second-level labeling scenario (the second-level labeling scenario needs to output a second-level label for each second of video, and the one-dimensional arrays overlap with each other, which can improve the recognition stability); for the fragment-level labeling scenario, when selecting the sliding window step and window size, the sliding window step can be set to be the same as the window size, so that the one-dimensional arrays after the slice processing do not overlap with each other, thereby adapting to the fragment-level labeling scenario (the fragment-level labeling scenario only needs to determine whether there are highlights in the fragment, so for the fragment-level labeling scenario, the one-dimensional arrays can be set not to overlap with each other).

[0087] In addition, for the second-level labeling scenario, it should be noted that when adding the beginning and the end of the sliced ​​one-dimensional array, it is necessary to ensure that each sliced ​​audio corresponding to each second of audio data must contain the audio data corresponding to that second in the original audio data, so as to ensure that the audio label of each second of audio data obtained based on each sliced ​​audio is accurate.

[0088] After obtaining the sliced ​​audio, feature extraction may be performed on the obtained sliced ​​audio to obtain Mel spectrum features.

[0089] Further, in one embodiment, extracting features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features includes:

[0090] Performing pre-emphasis processing on the sliced ​​audio to obtain first audio data;

[0091] Performing frame division and windowing operations on the first audio data to obtain second audio data;

[0092] Performing discrete Fourier transform on the second audio data to obtain frequency domain data;

[0093] The frequency domain data is identified using a Mel filter to obtain Mel spectrum features.

[0094] Specifically, the Mel feature extraction algorithm, also known as the Mel spectrum extraction technology, is based on the fact that the sound frequency perceived by the human ear is not linear with the actual frequency of the sound. By constructing a set of filters, the input signal is filtered, and the signal energy output by each filter is used as the basic feature of the signal. After further processing, the feature is used as the input feature of the speech. The Mel feature extraction algorithm does not depend on the nature of the signal, does not make any assumptions or restrictions on the input signal, and utilizes the research results of the auditory model. Therefore, it has good robustness, conforms to the auditory characteristics of the human ear, and still has good recognition performance when the signal-to-noise ratio is reduced.

[0095] Using the Mel feature extraction algorithm, the sliced ​​audio can be converted into a two-dimensional feature matrix with time domain and frequency domain information.

[0096] Specifically, based on the Mel feature extraction algorithm, the sliced ​​audio can be pre-emphasized first. When the sliced ​​audio is pre-emphasized, the high-frequency part of the sliced ​​audio can be enhanced using a high-pass filter. In addition, it is necessary to ensure that the spectrum of the sliced ​​audio is obtained using the same signal-to-noise ratio in the entire frequency band from low frequency to high frequency. Here, during actual enhancement, the high-frequency part of the sliced ​​audio can be enhanced by increasing the pre-emphasis coefficient.

[0097] In the present application, since the frequencies of highlights such as laughs, applause, and cheers are generally higher than the frequencies of other sounds. Therefore, in the present embodiment, the high-frequency part in the sliced ​​audio is enhanced to strengthen the detection of highlights in the present embodiment. For example, in a comedy variety show, in order to identify laughs, applause, and other points in the audio data, since the frequencies of laughs, applause, and other points are generally higher than other points, in order to achieve a better recognition effect of laughs, applause, and other points, after slicing the audio data, the 2000-3000Hz frequency band in the sliced ​​audio can be enhanced, and the pre-emphasis coefficient can be set to 0.97 to enhance the high-frequency part in the sliced ​​audio.

[0098] After the sliced ​​audio is pre-emphasized, the first audio data after the pre-emphasis processing can be framed to perform framing in the time domain. That is, the first audio data is divided into multiple one-dimensional first arrays according to a fixed time length. Here, each one-dimensional first array divided can be called a frame. Here, it should be noted that the framing operation in this embodiment does not refer to the slicing operation in the above embodiment. In this embodiment, the original audio data can be sliced ​​to obtain sliced ​​audio, and then the sliced ​​audio can be framed using the framing operation in the Mel feature extraction algorithm. After the framing operation, the multiple one-dimensional first arrays after framing can be windowed to increase the continuity of the left and right ends of the frame and reduce spectrum leakage. In actual application, a traditional window function can be used for windowing, for example, a Hamming window function.

[0099] After the windowing operation is performed, the second audio data after the windowing process can be subjected to a discrete Fourier transform, and the signal originally represented in the time domain can be transformed into a representation of energy in the frequency domain. Different energy distributions can represent different speech features. Here, the data after the discrete Fourier transform can be defined as frequency domain data.

[0100] After obtaining the frequency domain data, the frequency domain data can be identified using a Mel filter to obtain Mel spectrum features. Specifically, the frequency domain data can be passed into the Mel filter, and the linear spectrum can be mapped to a Mel nonlinear spectrum based on auditory perception to simulate auditory perception and obtain a two-dimensional Mel spectrum feature. Since the present application performs Mel spectrum feature recognition on sliced ​​audio in seconds, the identified Mel spectrum features can also be called second-level Mel spectrum features.

[0101] Further, in one embodiment, the inputting the original audio data and the Mel frequency spectrum feature into a preset deep model for audio recognition, and determining the audio label corresponding to the audio data per second in the original audio data includes:

[0102] Using a shallow feature extraction module in the deep model to extract features from the original audio data, to obtain a first feature;

[0103] Performing data enhancement on the Mel frequency spectrum feature using the deep model to obtain a second feature;

[0104] The first feature and the second feature are subjected to feature processing in the deep model to determine an audio label corresponding to each second of audio data in the original audio data.

[0105] Here, the deep model can be a convolutional neural network model. The deep model can dynamically adjust the convolution kernel weights according to the task objectives and penalty functions, perform feature extraction and classification mapping on the samples, generalize to the data samples in the business scenarios, and complete the label determination of the samples in the business scenarios.

[0106] Specifically, after receiving the original audio data and Mel spectrum features, the deep model first sends the original audio data to a shallow feature extraction model composed of a convolution layer and a pooling layer, and obtains some audio features ignored by the Mel spectrum through learnable parameters, namely the first feature. This feature can retain more information for the input of the deep feature map of the subsequent deep model. At the same time, the deep model can also perform data enhancement on the Mel spectrum features to obtain the second feature. Performing data enhancement on the Mel spectrum features can enhance the generalization of the network and prevent overfitting. Here, data enhancement methods such as Mixup and SpecAugment can be used.

[0107] In this embodiment, by using the deep model to perform second-level audio label inference operations on the original audio data and the Mel spectrum features, the time point of the highlight audio can be inferred. That is, in this embodiment, by using the deep model, it is possible to learn whether there is a sound label related to the audio highlight at present from the original audio data and the Mel spectrum features.

[0108] Further, in one embodiment, the performing feature processing on the first feature and the second feature in the deep model to determine the audio tag corresponding to the audio data per second in the original audio data includes:

[0109] Connecting the first feature and the second feature respectively to obtain a three-dimensional feature map;

[0110] Using a one-dimensional convolutional layer to extract features from the three-dimensional feature map in the time domain dimension to obtain three-dimensional time domain features in the time domain dimension;

[0111] Using a two-dimensional convolutional layer to extract features of the three-dimensional time domain features in the time domain and space dimensions, to obtain three-dimensional time domain and space features in the time domain and space dimensions;

[0112] Performing feature redundancy reduction and feature size unification processing on the three-dimensional time domain and space features to obtain uniform size features;

[0113] The uniform size features are spatially mapped and label identified to obtain an audio label corresponding to each second of audio data in the original audio data.

[0114] Specifically, when the first feature and the second feature are connected respectively, the first feature and the second feature can be connected respectively by a connection function to obtain a three-dimensional feature map. Connecting the first feature and the second feature can be understood as stacking the two-dimensional feature maps, and the stacked two-dimensional feature map is the three-dimensional feature map. Here, the connection function can be a concat() function.

[0115] After obtaining the three-dimensional feature map, the three-dimensional feature map can be passed to the one-dimensional convolution layer in the deep model. This layer is used to extract different features of the three-dimensional feature map in the time domain dimension to obtain the three-dimensional time domain features. After that, the three-dimensional time domain features can be sent to the two-dimensional convolution layer in the deep model to increase the receptive field and produce the features that integrate the time domain and space, that is, the three-dimensional time domain and space features in the time domain and space dimensions. After obtaining the three-dimensional time domain and space features, the three-dimensional time domain and space features can be sent to the global pooling layer in the deep model to reduce the redundancy of the features; at the same time, the different sizes in the three-dimensional time domain and space features are uniformly processed through the feature size to obtain a uniform size feature with a fixed size. After obtaining the uniform size feature, the uniform size feature can be sent to the fully connected layer in the deep model to map the learned "distributed feature representation" to the sample label space; and the result is output. After obtaining the output result, the output result can be sent to the Dropout layer in the deep model, where the Dropout layer will randomly discard some feature map content to prevent overfitting of the model. Finally, we can use the same number of discriminators as labels to predict the output of the Dropout layer to predict whether the sample has audio content of a certain label. Here, we can use the sigmoid discriminator for prediction.

[0116] Further, in one embodiment, the inputting the original audio data and the Mel-spectrogram feature into a preset deep model for audio recognition, and determining the audio label corresponding to the audio data per second in the original audio data further includes:

[0117] Obtaining training data and real audio labels corresponding to the training data;

[0118] Using the training data and the real audio labels to train the initial deep model to obtain a trained deep model;

[0119] The original audio data and the Mel-spectrogram feature are input into the trained deep model for audio recognition, and the audio label corresponding to each second of audio data in the original audio data is determined.

[0120] In actual application, before using the deep model for recognition and judgment, you can first build a deep model and train the deep model to obtain a deep model that can recognize highlights.

[0121] Here, the construction and training of the deep model can be achieved through the steps of audio data organization, model construction, pre-trained weight loading, multi-epoch training and weight preservation. Specifically, the training data is obtained first, that is, the original audio data and Mel spectrum features used for model training are obtained first. Here, the original audio data and Mel spectrum features used for model training can be organized into HDF5 format files, and the corresponding real audio labels can be obtained. Then, the deep model is built, and the open source AudioSet pre-trained weights are loaded into the built deep model as initialization weights to guide subsequent training. After the weights are loaded into the deep model, the HDF5 format file can be input into the deep model for multiple rounds of training. After the training is completed, the trained network weights can be stored on disk for subsequent inference process calls.

[0122] After obtaining the trained network weights, the trained network weights can be used to identify and judge audio highlights. Here, the identification and judgment of audio highlights can be achieved through weight loading, pre-processed data reading, forward reasoning and other steps. Specifically, the trained network weights are loaded into the deep model, and the original audio data to be detected is obtained, and the original audio data to be detected is input into the loaded deep model, and forward reasoning is performed to obtain a two-dimensional second-level label.

[0123] Further, in one embodiment, performing spatial mapping and label discrimination on the uniform size feature to obtain an audio label corresponding to each second of audio data in the original audio data includes:

[0124] Performing spatial mapping on the uniform size feature to obtain a label value corresponding to the uniform size feature;

[0125] Get the set recall threshold;

[0126] Compare the label value with the recall threshold to determine the label binary value corresponding to the label value;

[0127] It is determined according to the binary value of the label whether the audio data per second corresponding to the label value has the audio label corresponding to the binary value of the label.

[0128] Specifically, after spatial mapping of the uniform size feature, the label value corresponding to the feature can be obtained. That is, through this deep model, the label values ​​of multiple label categories can be obtained for 1 second of audio data. For the label value of a certain label category among multiple label categories, the recall threshold can be used to determine whether the second audio data has this type of label. When the label value of this type of label category is greater than or equal to the recall threshold, it can be determined that the second audio data has this type of label, and when the label value of this type of label category is less than the recall threshold, it can be determined that the second audio data does not have this type of label. Here, the recall threshold can be set based on the business scenario. According to different business needs, the parameters are flexibly adjusted to adapt the calculation results of the model to different application scenarios. In actual application, the label binarization value can be used to characterize whether the second audio data has this type of label. For example, the label binarization value can be set to 1 to represent that the second audio data has this type of label; the label binarization value can be set to 0 to represent that the second audio data does not have this type of label.

[0129] Further, in one embodiment, performing highlight generation processing on the audio tag corresponding to each second of audio data in the original audio data to obtain audio highlight segment comprises:

[0130] Get the smoothing step size and smoothing window size;

[0131] Slide the original audio data according to the smoothing step size, and perform numerical unification processing on the binary values ​​of the original audio data of the smoothing window size each time the data is slid, to obtain the binary values ​​after the numerical unification;

[0132] Slide the original audio data multiple times according to the smoothing step length to obtain a binary value after the audio data value per second in the original audio data is unified;

[0133] According to the continuity of the binary values ​​of the audio data values ​​per second in the original audio data after being unified, the audio segment corresponding to the binary values ​​of the continuous preset values ​​is determined as the audio highlight segment.

[0134] Specifically, the audio tags of the audio data per second can be smoothed, so as to generate a highlight segment with a time point according to the smoothed result. In actual application, the smoothing window size and smoothing step size can be formulated based on the business scenario and the continuity of the sound, and the determined binary values ​​can be connected into a smooth segment binary value, and the audio highlight segment can be determined according to the continuity of the smoothed binary value. For example, when the binary value of the audio data per second in the 9-second original audio data is 111011000, the smoothing window size and smoothing step size can be used to smooth the binary value to 111111000, then the audio segment from the 1st second to the 6th second in the original audio data is the audio highlight segment.

[0135] The audio highlight determination method provided by the embodiment of the present invention obtains original audio data; slices the original audio data to obtain sliced ​​audio, wherein the sliced ​​audio is in seconds; extracts features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features; inputs the original audio data and the Mel spectrum features into a preset deep model for audio recognition to determine the audio tags corresponding to the audio data per second in the original audio data; performs highlight generation processing on the audio tags corresponding to the audio data per second in the original audio data to obtain audio highlight segments. The solution provided by the present invention can be combined with the needs of business scenarios to complete automatic detection and identification of audio highlights.

[0136] The present invention is further described in detail below in conjunction with application examples.

[0137] This embodiment can be applied in the scenario where audio exists, to process and analyze the audio signal, analyze the time points of laughter, applause, and cheers in the audio, and perform superposition post-processing to complete automatic audio / video highlight point detection, such as: variety show golden sentence positioning, film and television drama highlight marking, etc. This embodiment can analyze and locate the sounds that may be the highlight points in the audio under the condition that samples are difficult to collect, combined with the business demand for finding highlights.

[0138] This embodiment mainly uses traditional speech feature extraction technology to characterize audio digital signals as digital features; and transmits the digital features into a deep network model, i.e., a supervised training model, to extract feature representations that can be used to distinguish interesting audio (laughter / applause / cheers), and sends the extracted feature representations to a sigmoid discriminator (which can be understood as a label discriminator) to achieve probability prediction of whether the audio is interesting or not on a second-by-second basis; then, a post-processing strategy suitable for the business scenario is used to smooth the second-level labels into segment-level labels, and output them as labels for subsequent business use.

[0139] Before introducing the specific solutions of this embodiment, several algorithm principles used in this embodiment are first introduced:

[0140] The first one: traditional Mel spectrum extraction technology.

[0141] Mel spectrum extraction technology is based on the fact that the sound frequency perceived by the human ear is not linear with the actual frequency of the sound. By constructing a set of filters, the input signal is filtered. The signal energy output by each filter is used as the basic feature of the signal. After further processing, this feature can be used as the input feature of the speech. Since this feature does not depend on the nature of the signal, it does not make any assumptions or restrictions on the input signal, and utilizes the research results of the auditory model. Therefore, this feature has good robustness, is more in line with the auditory characteristics of the human ear, and still has good recognition performance when the signal-to-noise ratio is reduced.

[0142] The second type: deep model, mainly convolutional neural network.

[0143] The main principle is to dynamically adjust the convolution kernel weights according to the task objectives and penalty functions, perform feature extraction and classification mapping on the samples, generalize to the data samples in the business scenarios, and complete the label determination of the samples in the business scenarios.

[0144] Specifically, the solution proposed in this embodiment is mainly divided into three modules: data audio signal preprocessing module, second-level tag recognition module, and post-processing module based on business scenarios. That is, the specific operation process steps of this embodiment can be as follows: Figure 2 As shown.

[0145] S1: Use the data audio signal preprocessing module to load the audio and process it into Mel spectrum features.

[0146] S2: The original signal array and Mel spectrum features generated in S1 are fed into the deep model to perform second-level audio label inference operations to infer the time location of the interesting audio.

[0147] S3: The second-level labels output by S2 are connected into highlight segments with time points through the post-processing module.

[0148] Below, the above steps will be explained in further detail.

[0149] (I) Data audio signal preprocessing module

[0150] In this embodiment, the pre-processing of the digital signal mainly utilizes the traditional Mel feature extraction algorithm to convert the digital signal into a two-dimensional feature matrix with time domain and frequency domain information.

[0151] Specifically, see Figure 3 ,The operation process of the Mel spectrum extraction algorithm includes the following:

[0152] The Mel feature extraction algorithm first performs a pre-emphasis operation on the loaded voice data, passes the voice signal through a high-pass filter, enhances the high-frequency part of the voice signal, and keeps it in the entire frequency band from low frequency to high frequency, so that the spectrum can be obtained using the same signal-to-noise ratio. In this embodiment, since the audio of the highlights is detected according to the business scenario involved, the pre-emphasis coefficient can be set larger to highlight the high-frequency audio information. Next, the one-dimensional audio data is framed in the time domain, and the one-dimensional number is divided into multiple one-dimensional arrays according to a fixed time length, where each one-dimensional array is called a frame; then, each frame is windowed, and the window function used in this embodiment can be a traditional Hamming window to increase the continuity of the left and right ends of the frame and reduce spectrum leakage. After the windowing process, a frame of audio uses a discrete Fourier transform to convert the signal originally represented in the time domain into a representation of energy in the frequency domain. Different energy distributions can represent different speech features. Next, the frequency domain data is passed into the Mel filter, which maps the linear spectrum to the Mel nonlinear spectrum based on auditory perception, simulates auditory perception, and finally obtains a two-dimensional Mel spectrum feature.

[0153] Based on the operation process of the above-mentioned Mel feature extraction algorithm, the specific working steps of the digital signal preprocessing module in this embodiment can be as follows: Figure 4 As shown. Includes the following:

[0154] S11: Use ffmpeg (which can be understood as an audio separation technology) to separate the target audio from the audio and video samples and re-encode them into wav format; use the open source audio processing library librosa to load the audio signal into the memory and save it in a one-dimensional array;

[0155] S12: The original audio data output by S11 is sliced ​​and preprocessed according to the business scenario;

[0156] S13: extract Mel spectrum features from the sliced ​​audio output by S12 respectively, and understand them as Mel features of the current second.

[0157] Here, the detailed process of S12 can be mainly divided into the following two steps:

[0158] S121: Based on the detection granularity requirements of the business scenario, set the sliding window step size; based on the subsequent deep model, the detection confidence and sample length characteristics in the scenario, set the window size; use the step size and window size to split the original one-dimensional audio data into multiple one-dimensional arrays;

[0159] S122: Using the copy and expansion method, the number of one-dimensional single-chip audio arrays output by the S121 strategy is supplemented at the head and tail to make the number the same as the original audio seconds. The subsequent calculation results of the current audio chip array represent the calculation results of the current seconds.

[0160] Here, the sliding window step size and window size can be selected based on the needs of the business scenario. For example, for highlights such as applause, since applause is similar to the sound of rain, in order to accurately identify the applause and avoid interference from sounds such as rain, the window size can be set to a larger value, such as 3 seconds. By setting the window size to a larger value, the sliced ​​audio after slicing is made longer, so that the accuracy of identifying applause can be improved during recognition, and interference from rain sounds can be avoided. For another example, for highlights such as laughter, since laughter is very different from other sounds and is easier to identify, here, the window size can be selected as 1 second to achieve the detection of highlights of laughter.

[0161] In addition, since the present application is aimed at identifying second-level tag scenarios, the sliding window step size can be set to 1 second to ensure better recognition results.

[0162] Specifically, the original audio data can be slid according to the sliding window step size, and each time it slides, the original audio data after sliding is divided according to the window size to obtain a one-dimensional array; the original audio data is slid multiple times according to the sliding window step size to obtain multiple one-dimensional arrays. For example, based on the detection granularity requirements of the business scenario, the sliding window step size can be set to 1 second; then based on the characteristics of the deep model, the detection confidence in the business scenario and / or the sample length, the window size is set to 2 seconds. Then, when the original audio is 10 seconds, the original audio data is slid at a speed of sliding 1 second of the original audio data each time, and each time it slides, a 2-second audio data is sliced ​​out from the original audio data after sliding, and each sliced ​​2-second audio data is a one-dimensional array. Slicing 10 seconds of audio data according to a sliding window step size of 1 second and a window size of 2 seconds can obtain a total of 9 one-dimensional arrays.

[0163] After slicing, the number of one-dimensional single-chip audio arrays output by the S121 strategy is supplemented at the head and tail to make the number the same as the number of seconds of the original audio. For example, when the sliding window step size is 1 second and the window size is 2 seconds, 10 seconds of original audio data is sliced ​​to obtain 9 one-dimensional arrays. The first one-dimensional array can be copied at the head of the 9 one-dimensional arrays, or the last one-dimensional array can be copied at the tail of the 9 one-dimensional arrays to make the number of the one-dimensional arrays the same as the number of seconds of the original audio.

[0164] The above is the relevant content of the data audio signal preprocessing module in this embodiment.

[0165] (II) Second-level tag recognition module

[0166] In this module, we mainly build a deep model (AudioCNN) and a supervised training model so that the model can learn from the original audio features (Mel spectrum features) whether there are sound labels related to the current audio highlights.

[0167] Here, the specific network model can be Figure 5 As shown, it includes the following processes:

[0168] like Figure 5 As shown in the figure, the network model will simultaneously receive the original audio data and the Mel spectrum features produced by the digital signal preprocessing module as the input of the AudioCNN model. The original audio data will be sent to a shallow feature extraction model consisting of a convolutional layer and a pooling layer, in order to obtain some audio features ignored by the Mel spectrum through learnable parameters, so as to retain more information for the input of subsequent deep feature mappings; and the other Mel feature input branch will perform effective data enhancement after obtaining the Mel spectrum features, including: Mixup data enhancement and SpecAugment data enhancement, in order to enhance the generalization of the network and prevent overfitting. After completing the above processing, the features produced by different input channels are connected into a three-dimensional feature map using the concat method, and then passed to the one-dimensional convolution layer to extract features in different time domains; then, two-dimensional convolution is used to increase the receptive field and produce features that integrate time domain and space; then, it is passed to the global pooling layer to reduce feature redundancy. At the same time, feature maps of different sizes generated by inputs of different lengths are fixed to a specific dimension; the features of unified dimension produced by the pooling layer will be sent to the fully connected layer to map the learned "distributed feature representation" to the sample label space; the output of each fully connected layer will be sent to the Dropout layer to randomly discard some featuremap content to prevent overfitting of the model; finally, the same number of sigmoid activators as the label are used to predict whether the sample has audio content of a certain label.

[0169] Based on the above deep model, the main process of this module may include:

[0170] like Figure 6 As shown: This second-level tag recognition module can be mainly divided into the following two processes: training process and use process.

[0171] (1) Training process. Mainly includes: audio data organization, model building, pre-training weight loading, multi-epoch training and weight preservation. Specifically:

[0172] S21: After the training data is processed by the data audio signal preprocessing module, the output raw audio data and Mel spectrum features are organized into an HDF5 format file, which records the data and the corresponding real audio labels.

[0173] S22: Build the above-mentioned deep model.

[0174] S23: Load the open source AudioSet pre-trained weights into the deep model built in step S22 as initialization weights to guide subsequent training.

[0175] S24: Input the data prepared in step S21 into the deep model initialized in step S23 and perform multiple rounds of training.

[0176] S25: After step S24 is completed, the network weights are stored in the disk for subsequent inference process calls.

[0177] (2) Usage process. Mainly includes: weight loading, pre-processing data reading, and forward reasoning. Specifically:

[0178] S26: Load the weights stored in step S25 into the model.

[0179] S27: After being processed by the data audio signal preprocessing module, the audio data to be detected is read.

[0180] S28: Send the data read in step S27 to the model prepared in step S26 for forward reasoning.

[0181] S29: Produce two-dimensional second-level labels for use by subsequent post-processing modules.

[0182] The above is the relevant content of the second-level tag recognition module in this embodiment.

[0183] (III) Post-processing module

[0184] The post-processing module in this embodiment is mainly designed for business scenario requirements. In addition, according to different business requirements, parameters can be flexibly adjusted to make the calculation results of the model adapt to different application scenarios.

[0185] Specifically, see Figure 7 The overall process design of the post-processing module in this embodiment is mainly divided into the following two steps: first, the recall mechanism design of the second-level label, and then the sliding window design of the second-level label into the fragment-level label. The above two steps are specifically:

[0186] (1) Based on the business scenario, the original tags are divided into highlight tags. Then, through business guidance examples, the recall threshold of the highlight tags is determined to produce binary second-level tags.

[0187] (2) Analyze the business scenario and the continuity of the sound, determine the smoothing window size, connect the binary second-level labels into smooth segment-level labels, and finally produce an audio highlight guidance segment.

[0188] Specifically, the above-mentioned deep model will output the label values ​​of multiple label categories corresponding to each second of audio data in the original audio data. Using the determined recall threshold, the label value is judged to determine whether the second audio data has the label of this category. Here, the judgment result is represented by a binary value. For example, when the binary value is 1, it indicates that the second audio data has the label of this category; when the binary value is 0, it indicates that the second audio data does not have the label of this category.

[0189] Then, based on the business scenario, a smoothing step size and a smoothing window size are set, and the original audio data is slid according to the smoothing step size. Each time the data is slid, the binary values ​​of the original audio data of the smoothing window size are numerically unified to obtain the unified binary values. The original audio data is slid multiple times according to the smoothing step size to obtain the unified binary values ​​of the audio data per second in the original audio data. According to the continuity of the unified binary values ​​of the audio data per second in the original audio data, the audio segment corresponding to the binary values ​​of the continuous preset values ​​is determined to be the audio highlight segment.

[0190] For example, when the binarized value of the audio data per second in 9 seconds of original audio data is 111011000, the binarized value can be smoothed to 111111000 using the smoothing window size and smoothing step size, then the audio segment from the 1st second to the 6th second in the original audio data is the audio highlight segment.

[0191] By designing a label recall mechanism and smoothing processing, audio highlights can be automatically generated in the original audio data.

[0192] The above is the relevant content of the post-processing module in this embodiment.

[0193] In summary, after the relevant contents of the above three modules of this embodiment are introduced in detail, the complete process of this embodiment may include:

[0194] Specifically, see Figure 8As described above, the system of this embodiment is mainly divided into three modules: a data audio signal preprocessing module, a second-level label recognition module and a business scenario post-processing module. Among them, the audio signal preprocessing module executes the process of loading the audio signal into the memory, acquiring one-dimensional audio data, Mel spectrum extraction, and two-dimensional Mel features; the second-level label recognition module executes the process of data enhancement, model training, model forward reasoning, and second-level label recognition; the business scenario post-processing module executes the process of second-level label binarization and smoothing into fragment labels. Through the above three modules, the highlight audio fragment is finally obtained. This implementation sends the original audio into the system, which can realize automatic detection and write out the audio highlight guidance fragment for subsequent business logic superposition and use.

[0195] In addition, in summary, compared with the prior art, this embodiment has the following advantages: (1) In audio preprocessing, the high-frequency frequency band spectrum is amplified according to the characteristics of the highlight audio, so that its features are easier to be learned by the model. (2) In the deep model, the shallow original audio feature extraction layer is embedded, which can better learn the diversity of audio features, so that it is not limited to the feature information provided by the Mel spectrum, thereby enhancing the generalization of the network. (3) The post-processing is parameterized, and the reference parameter range can be automatically given without too much manual intervention. (4) Combining audio tags with highlight requirements, the audio highlight tags are proposed.

[0196] This embodiment combines traditional feature processing with data enhancement to compensate for the effective training of the supervision model when data samples are insufficient; at the same time, in view of the task positioning requirements, the existing model is improved to make it more reliable in analyzing short-term audio scenarios; finally, in view of the business scenarios and target sound characteristics, a pre-processing method for expanding specific frequency bands and a post-processing device for controllable interval smoothing are proposed. This embodiment is applied to the highlight editing business and pure enjoyment business, and can automatically provide the highlight and wonderful point information from the audio perspective for the selection of later editing, reducing labor costs.

[0197] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides an audio highlight point determination device, such as Fig. 9 As shown, the audio highlight determination device 900 includes: an acquisition module 901, a slicing module 902, a feature extraction module 903, an audio recognition module 904 and a highlight generation module 905; wherein,

[0198] An acquisition module 901 is used to acquire original audio data;

[0199] The slicing module 902 is used to slice the original audio data to obtain sliced ​​audio, where the sliced ​​audio is in seconds;

[0200] A feature extraction module 903 is used to extract features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features;

[0201] An audio recognition module 904 is used to input the original audio data and the Mel-spectrogram feature into a preset deep model for audio recognition, and determine an audio label corresponding to each second of audio data in the original audio data;

[0202] The highlight generation module 905 is used to perform highlight generation processing on the audio tags corresponding to the audio data per second in the original audio data to obtain audio highlight segments.

[0203] In actual application, the acquisition module 901, the slicing module 902, the feature extraction module 903, the audio recognition module 904 and the highlight point generation module 905 can be implemented by a processor in the audio highlight point determination device.

[0204] It should be noted that: when the above-mentioned device provided in the above-mentioned embodiment is executed, only the division of the above-mentioned program modules is used as an example. In actual application, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the terminal is divided into different program modules to complete all or part of the above-mentioned processing. In addition, the above-mentioned device provided in the above-mentioned embodiment and the above-mentioned method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0205] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides an electronic device (computer device). Specifically, in one embodiment, the computer device can be a terminal, and its internal structure diagram can be as follows: Fig.10As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0206] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0207] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0208] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0209] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0210] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0211] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0212] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0213] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0214] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0215] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0216] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0217] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for determining audio highlights, characterized in that: The method comprises: Get the original audio data; Slicing the original audio data to obtain sliced ​​audio, where the sliced ​​audio is in seconds; Extracting features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features; Input the original audio data and the Mel-spectrogram feature into a preset deep model for audio recognition, and determine the audio label corresponding to the audio data per second in the original audio data; Performing a highlight generation process on the audio tag corresponding to each second of the audio data in the original audio data to obtain an audio highlight segment; The step of inputting the original audio data and the Mel spectrum feature into a preset deep model for audio recognition and determining the audio tag corresponding to the audio data per second in the original audio data comprises: performing feature extraction on the original audio data using a shallow feature extraction module in the deep model to obtain a first feature; performing data enhancement on the Mel spectrum feature using the deep model to obtain a second feature; performing feature processing on the first feature and the second feature in the deep model to determine the audio tag corresponding to the audio data per second in the original audio data; Among them, the feature processing of the first feature and the second feature in the deep model to determine the audio label corresponding to the audio data per second in the original audio data includes: connecting the first feature and the second feature respectively to obtain a three-dimensional feature map; using a one-dimensional convolutional layer to perform feature extraction on the three-dimensional feature map in the time domain dimension to obtain a three-dimensional time domain feature in the time domain dimension; using a two-dimensional convolutional layer to perform feature extraction on the three-dimensional time domain feature in the time domain and space dimensions to obtain three-dimensional time domain and space features in the time domain and space dimensions; reducing feature redundancy and unifying feature size of the three-dimensional time domain and space features to obtain uniform size features; performing spatial mapping and label discrimination on the uniform size features to obtain the audio label corresponding to the audio data per second in the original audio data.

2. The method according to claim 1, characterized in that The slicing of the original audio data to obtain the sliced ​​audio includes: Get the set sliding window step size and window size; According to the sliding window step size and the window size, the original audio data is divided into a plurality of one-dimensional arrays; Obtain the number of the one-dimensional arrays and the number of seconds of the original audio data; According to the number of the plurality of one-dimensional arrays and the number of seconds of the original audio data, a preset number of first arrays and / or second arrays are copied at the head and the tail of the plurality of one-dimensional arrays, so that the number of the one-dimensional arrays is the same as the number of seconds of the original audio data; wherein the first array is the first one-dimensional array obtained when the original audio data is segmented; and the second array is the last one-dimensional array obtained when the original audio data is segmented; The multiple one-dimensional arrays copied head to tail are used as the sliced ​​audios of the original audio data.

3. The method according to claim 1, characterized in that The extracting features of the sliced ​​audio by using a Mel feature extraction algorithm to obtain Mel spectrum features comprises: Performing pre-emphasis processing on the sliced ​​audio to obtain first audio data; Performing frame division and windowing operations on the first audio data to obtain second audio data; Performing discrete Fourier transform on the second audio data to obtain frequency domain data; The frequency domain data is identified using a Mel filter to obtain Mel spectrum features.

4. The method according to claim 1, characterized in that: The performing of spatial mapping and label discrimination on the uniform size feature to obtain the audio label corresponding to the audio data per second in the original audio data comprises: Performing spatial mapping on the uniform size feature to obtain a label value corresponding to the uniform size feature; Get the set recall threshold; Compare the label value with the recall threshold to determine the label binary value corresponding to the label value; It is determined according to the binary value of the label whether the audio data per second corresponding to the label value has the audio label corresponding to the binary value of the label.

5. The method according to claim 4, characterized in that The step of performing a highlight generation process on the audio tag corresponding to each second of the audio data in the original audio data to obtain an audio highlight segment comprises: Get the smoothing step size and smoothing window size; Slide the original audio data according to the smoothing step size, and perform numerical unification processing on the binary values ​​of the original audio data of the smoothing window size each time the data is slid, to obtain the binary values ​​after the numerical unification; Slide the original audio data multiple times according to the smoothing step length to obtain a binary value after the audio data value per second in the original audio data is unified; According to the continuity of the binary values ​​of the audio data values ​​per second in the original audio data after being unified, the audio segment corresponding to the binary values ​​of the continuous preset values ​​is determined as the audio highlight segment.

6. An audio highlight determination device, characterized in that: The audio highlight point determination device comprises: An acquisition module, used to acquire original audio data; A slicing module, used for slicing the original audio data to obtain sliced ​​audio, where the sliced ​​audio is in seconds; A feature extraction module, used to extract features from the sliced ​​audio using a Mel feature extraction algorithm to obtain Mel spectrum features; An audio recognition module, used to input the original audio data and the Mel spectrum features into a preset deep model for audio recognition, and determine the audio label corresponding to the audio data per second in the original audio data; A highlight generation module is used to perform highlight generation processing on the audio tags corresponding to the audio data per second in the original audio data to obtain audio highlight segments; The audio recognition module is further used to: use the shallow feature extraction module in the deep model to perform feature extraction on the original audio data to obtain a first feature; use the deep model to perform data enhancement on the Mel spectrum feature to obtain a second feature; perform feature processing on the first feature and the second feature in the deep model to determine the audio label corresponding to the audio data per second in the original audio data; wherein the feature processing on the first feature and the second feature in the deep model to determine the audio label corresponding to the audio data per second in the original audio data includes: connecting the first feature and the second feature respectively to obtain a three-dimensional feature map; using a one-dimensional convolutional layer to perform feature extraction on the three-dimensional feature map in the time domain dimension to obtain a three-dimensional time domain feature in the time domain dimension; using a two-dimensional convolutional layer to perform feature extraction on the three-dimensional time domain feature in the time domain and space dimensions to obtain three-dimensional time domain and space features in the time domain and space dimensions; performing feature redundancy reduction and feature size unification processing on the three-dimensional time domain and space features to obtain a unified size feature; performing spatial mapping and label discrimination on the unified size feature to obtain an audio label corresponding to the audio data per second in the original audio data.

7. An electronic device, characterized in that: include: A processor and a memory for storing a computer program capable of running on the processor; wherein, When the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 5 are performed.

8. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images