Method, device and computer program product for identifying audio sensitive content

Abstract

The present application relates to the field of audio recognition, and provides a method, computer device, and computer program product for identifying audio-sensitive content. The present application can improve the accuracy of identifying audio-sensitive content. The method includes: first extracting a time-frequency spectrum of the audio to be identified, dividing the time-frequency spectrum into a plurality of time-frequency spectrum blocks, obtaining a plurality of block embedding features corresponding to the plurality of time-frequency spectrum blocks through a block embedding layer, and then inputting the plurality of block embedding features and their segmentation order information into a position embedding layer to obtain a position embedding feature sequence output by the position embedding layer, the position embedding feature sequence containing each block embedding feature and its position feature in the time-frequency spectrum, and then inputting the position embedding feature sequence into the encoder of the self-attention model to obtain a feature representation of the time-frequency spectrum, and finally identifying whether the audio to be identified contains sensitive content based on the feature representation of the time-frequency spectrum.

Description

Technical Field

[0001] The present application relates to the technical field of audio recognition, and in particular to a method for recognizing audio-sensitive content, a computer device, and a computer program product. Background Art

[0002] With the development of Internet technology, the forms of content dissemination have been enriched by uploading, publishing and disseminating audio and video works or live broadcasting of real-time audio and video. In order to ensure a good dissemination environment for high-quality content, it is necessary to accurately identify audio containing sensitive content.

[0003] The current technology mainly relies on artificially constructed features that are formed by the fusion of multiple audio features to identify audio-sensitive content, such as Mel-frequency cepstral coefficients, energy, zero-crossing rate, sub-band energy, sub-band energy entropy, spectrum center, spectrum extension, spectrum entropy, timbre deviation, etc., which are input into the neural network as feature vectors for classification and identification. However, the recognition accuracy of audio-sensitive content by this technology is easily restricted by artificially constructed features, and it will also cause some information loss, making its recognition accuracy of audio-sensitive content low. Summary of the invention

[0004] Based on this, it is necessary to provide a method for identifying audio sensitive content, a computer device and a computer program product to address the above technical issues.

[0005] In a first aspect, the present application provides a method for identifying audio-sensitive content. The method comprises:

[0006] Obtaining a time-spectrogram of the audio to be recognized, and dividing the time-spectrogram into a plurality of time-spectrogram blocks;

[0007] Obtaining block embedding features corresponding to each of the plurality of time-frequency spectrum blocks through a trained block embedding layer to obtain a plurality of block embedding features;

[0008] Input the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum blocks into the trained position embedding layer to obtain a position embedding feature sequence output by the position embedding layer; the position embedding features in the position embedding feature sequence include block embedding features and position features of the time-frequency spectrum blocks corresponding to the block embedding features in the time-frequency spectrum;

[0009] Inputting the position embedding feature sequence into an encoder of a trained self-attention model to obtain a feature representation of the time-spectrogram output by the encoder;

[0010] Based on the feature representation of the time-frequency spectrum graph, it is identified whether the audio to be identified contains sensitive content.

[0011] In one embodiment, dividing the time-frequency spectrum graph into a plurality of time-frequency spectrum graph blocks comprises:

[0012] The time-frequency spectrum diagram is segmented according to a preset segmentation order and a preset number of overlapping dimensions to obtain the multiple time-frequency spectrum diagram blocks; the preset number of overlapping dimensions includes the number of overlapping dimensions in the time dimension and / or the frequency dimension.

[0013] In one embodiment, the step of inputting the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum blocks into a trained position embedding layer includes: generating a plurality of segmentation order identifiers according to the segmentation order of the plurality of time-frequency spectrum blocks, and carrying the plurality of segmentation order identifiers in the plurality of block embedding features accordingly; and inputting the plurality of block embedding features carrying the plurality of segmentation order identifiers into the position embedding layer.

[0014] In one embodiment, the position embedding layer is used to obtain a plurality of position features corresponding to the plurality of block embedding features according to a plurality of segmentation order identifiers carried in the plurality of block embedding features, combine the corresponding block embedding features and position features to obtain a position embedding feature, and form the position embedding feature sequence according to the plurality of position embedding features obtained by combining.

[0015] In one embodiment, the identifying whether the audio to be identified contains sensitive content based on the feature representation of the time-spectrogram includes: inputting the feature representation of the time-spectrogram into a trained audio sensitive content classifier to obtain a sensitive content classification label output by the audio sensitive content classifier; and determining whether the audio to be identified contains sensitive content based on the sensitive content classification label.

[0016] In one embodiment, the method also includes: obtaining a pre-trained audio classification model, the audio classification model including a block embedding layer, a position embedding layer and an encoder of a self-attention model; obtaining a time-spectrogram sample of the audio sample and obtaining a sensitive content classification label corresponding to the audio sample; and using the time-spectrogram sample and the sensitive content classification label to train the embedding layer, position embedding layer, encoder of the self-attention model and the audio sensitive content classifier of the audio classification model.

[0017] In one embodiment, obtaining the sensitive content classification label corresponding to the audio sample includes: obtaining a first label of the audio sample based on whether the audio sample contains sensitive content; obtaining a second label of the audio sample based on a proportion of sensitive content in the audio sample; and using the first label and the second label as the sensitive content classification labels corresponding to the audio sample.

[0018] In one embodiment, the use of the time-spectrogram samples and the sensitive content classification labels to train the embedding layer, position embedding layer, encoder of the self-attention model and the audio sensitive content classifier of the audio classification model includes: performing masking processing on the time-spectrogram samples in the time dimension and / or frequency dimension to obtain a first interference sample of the time-spectrogram samples; performing mixing processing on the time-spectrogram samples in the time dimension and / or frequency dimension to obtain a second interference sample of the time-spectrogram samples; and using the time-spectrogram samples, the first interference sample, the second interference sample and the sensitive content classification labels to train the embedding layer, position embedding layer, encoder of the self-attention model and the audio sensitive content classifier of the audio classification model.

[0019] In a second aspect, the present application also provides a method for identifying audio-sensitive content. The method comprises:

[0020] Get the original audio;

[0021] Input the original audio into a first audio sensitive content recognition model to obtain multiple audio segments containing sensitive content in the original audio output by the first audio sensitive content recognition model; input the multiple audio segments as audio to be recognized into a second audio sensitive content recognition model to obtain sensitive content recognition results for each audio segment output by the second audio sensitive content recognition model; determine whether the original audio is sensitive audio based on the sensitive content recognition results of each audio segment; or,

[0022] Input the original audio as the audio to be identified into a second audio sensitive content recognition model, and obtain a sensitive content recognition result for the original audio output by the second audio sensitive content recognition model; when the sensitive content recognition result output by the second audio sensitive content recognition model indicates that the original audio is audio containing sensitive content, input the original audio into a first audio sensitive content recognition model, and obtain a sensitive content recognition result for the original audio output by the first audio sensitive content recognition model; determine whether the original audio is sensitive audio based on the sensitive content recognition result output by the first audio sensitive content recognition model; or,

[0023] Input the original audio into a first audio sensitive content recognition model to obtain a sensitive content recognition result for the original audio output by the first audio sensitive content recognition model; input the original audio as the audio to be recognized into a second audio sensitive content recognition model to obtain a sensitive content recognition result for the original audio output by the second audio sensitive content recognition model; determine whether the original audio is sensitive audio according to the sensitive content recognition result output by the first audio sensitive content recognition model and the sensitive content recognition result output by the second audio sensitive content recognition model;

[0024] The second audio sensitive content recognition model is used to identify whether the audio to be recognized contains sensitive content according to the method described above.

[0025] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0026] A time-frequency spectrum graph of the audio to be identified is obtained, and the time-frequency spectrum graph is divided into a plurality of time-frequency spectrum graph blocks; block embedding features corresponding to each of the plurality of time-frequency spectrum graph blocks are obtained through a trained block embedding layer to obtain a plurality of block embedding features; the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum graph blocks are input into a trained position embedding layer to obtain a position embedding feature sequence output by the position embedding layer; the position embedding features in the position embedding feature sequence include block embedding features and position features of the time-frequency spectrum graph blocks corresponding to the block embedding features in the time-frequency spectrum graph; the position embedding feature sequence is input into an encoder of a trained self-attention model to obtain a feature representation of the time-frequency spectrum graph output by the encoder; based on the feature representation of the time-frequency spectrum graph, whether the audio to be identified contains sensitive content is identified.

[0027] In a fourth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0028] A time-frequency spectrum graph of the audio to be identified is obtained, and the time-frequency spectrum graph is divided into a plurality of time-frequency spectrum graph blocks; block embedding features corresponding to each of the plurality of time-frequency spectrum graph blocks are obtained through a trained block embedding layer to obtain a plurality of block embedding features; the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum graph blocks are input into a trained position embedding layer to obtain a position embedding feature sequence output by the position embedding layer; the position embedding features in the position embedding feature sequence include block embedding features and position features of the time-frequency spectrum graph blocks corresponding to the block embedding features in the time-frequency spectrum graph; the position embedding feature sequence is input into an encoder of a trained self-attention model to obtain a feature representation of the time-frequency spectrum graph output by the encoder; based on the feature representation of the time-frequency spectrum graph, whether the audio to be identified contains sensitive content is identified.

[0029] The above-mentioned audio sensitive content recognition method, computer device and computer program product first extract the time-frequency spectrum of the audio to be recognized, divide the time-frequency spectrum into multiple time-frequency spectrum blocks, obtain multiple block embedding features corresponding to the multiple time-frequency spectrum blocks through the trained block embedding layer, and then input the multiple block embedding features and their segmentation order information into the trained position embedding layer to obtain the position embedding feature sequence output by the position embedding layer, which contains each block embedding feature and its position feature in the time-frequency spectrum. Then, the position embedding feature sequence is input into the encoder of the trained self-attention model to obtain the feature representation of the time-frequency spectrum, and finally, based on the feature representation of the time-frequency spectrum, it is identified whether the audio to be recognized contains sensitive content. This scheme extracts the local feature information and spatial structure information of the time-frequency spectrum through the self-learning of the block embedding layer and the position embedding layer to avoid information loss caused by artificial feature selection, and then the encoder of the self-attention model performs global feature learning of the time-frequency spectrum to obtain the feature representation of the time-frequency spectrum and finally identifies whether the audio to be recognized is audio containing sensitive content based on the feature representation, thereby improving the accuracy of audio sensitive content recognition. This solution can be applied to the identification of audio-sensitive content in the massive audio and video data generated by Internet products, significantly reducing the labor cost of review, and improving recognition efficiency while increasing recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of a method for identifying audio-sensitive content in one embodiment;

[0031] Figure 2 A schematic diagram of a process for processing audio to be recognized in one embodiment;

[0032] Figure 3 is a flowchart of a training step in an embodiment;

[0033] Figure 4is a flow chart of a method for identifying audio-sensitive content in another embodiment;

[0034] Figure 5 A schematic diagram of a flow chart of processing audio to be recognized in another embodiment;

[0035] Figure 6 is a flowchart of a method for identifying audio-sensitive content in yet another embodiment;

[0036] Figure 7 A schematic diagram of a flow chart of a method for identifying audio-sensitive content in yet another embodiment;

[0037] Figure 8 A schematic diagram of an audio-sensitive content recognition result interface in an application example;

[0038] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] The audio-sensitive content recognition method provided in the embodiment of the present application can be executed by computer devices such as terminals and servers. Among them, the terminal can be but is not limited to various personal computers, laptops, and tablet computers; the server can be implemented by an independent server or a server cluster composed of multiple servers. In terms of application scenarios, the audio-sensitive content recognition method provided in the present application can be specifically applied to products such as music applications, karaoke applications, and short video applications, and the audio-sensitive content of audio and video works uploaded, published, and disseminated in the product or live real-time audio and video can be accurately identified. This also facilitates relevant personnel to handle users involved in uploading, publishing, and disseminating audio-sensitive content, reducing the labor cost of sensitive content recognition for massive audio in the product, while improving recognition accuracy. Improve recognition efficiency and provide a good dissemination environment for high-quality content.

[0041] The following describes the audio-sensitive content recognition method provided by the present application in combination with various embodiments and corresponding drawings.

[0042] In one embodiment, a method for identifying audio-sensitive content is provided, combining Figure 1 and 2 , the method may include the following steps:

[0043] Step S101, obtaining a time-frequency spectrum diagram of the audio to be recognized.

[0044] The audio to be identified can be audio data uploaded, published and disseminated by users. The specific form of the audio data can be pure audio data uploaded, published and disseminated by users, or audio data in videos uploaded, published and disseminated by users. The duration of the audio to be identified can be of any duration, that is, the method for identifying audio-sensitive content provided in this application can identify sensitive content in audios of different durations. In this step, after obtaining the audio to be identified, extract the time-frequency spectrum of the audio to be identified. In the specific implementation, refer to Figure 2 For the input audio to be identified, a 128-dimensional fbank feature extraction can be performed, and a 25ms Hamming window and a 10ms frame shift are used to normalize the time-frequency features to obtain a time-frequency spectrum. The time-frequency spectrum contains the features of the audio time domain signal in the time-frequency domain after short-time Fourier transform. The horizontal axis represents time, the vertical axis represents frequency, and the depth of the color represents the energy / intensity of the signal at the corresponding time and frequency. This step extracts the original feature information of the audio to be identified, retains all the information representing the audio features, and does not lose any information at the source.

[0045] Step S102, dividing the time-frequency spectrum graph into a plurality of time-frequency spectrum graph blocks.

[0046] like Figure 2 In this step, the time-spectrogram of the audio to be recognized is segmented to obtain multiple time-spectrogram blocks. In a specific implementation, the time-spectrogram of the audio to be recognized can be segmented according to a set block size, such as segmenting the time-spectrogram of the audio to be recognized into N time-spectrogram blocks of each size of 16×16.

[0047] In one embodiment, step S102 specifically includes: dividing the time-frequency spectrum graph according to a preset division order and a preset overlapping dimension number to obtain a plurality of time-frequency spectrum graph blocks; the preset overlapping dimension number includes the overlapping dimension number in the time dimension and / or the frequency dimension.

[0048] In this embodiment, the preset segmentation order can adopt the row priority order for the time-spectrogram, that is, the time-spectrogram is segmented row by row, and the segmentation is performed according to the preset overlapping dimension number to maintain the continuity of the audio features of adjacent time-spectrogram blocks, thereby improving the accuracy of audio-sensitive content recognition. Among them, the preset overlapping dimension number includes the overlapping dimension number in the time dimension and / or frequency dimension. In the time-spectrogram, the row direction represents the time dimension, and the column direction represents the frequency dimension. Therefore, when the time-spectrogram is segmented in the row direction, it is necessary to make the two adjacent time-spectrogram blocks have the preset overlapping dimension number, and when the time-spectrogram is segmented in the column direction (that is, when the segmentation of one row is completed and the next row is entered), it is also necessary to make the time-spectrogram blocks of adjacent rows have the preset overlapping dimension number. In practical applications, when the time-spectrogram is segmented, the time-spectrogram can be cut into N time-spectrogram blocks of each size of 16×16, and the time dimension and frequency dimension have a preset overlapping dimension number of 6, that is, both time and frequency overlap in 6 dimensions.

[0049] Step S103, obtaining block embedding features corresponding to each of the plurality of time-frequency spectrum blocks through the trained block embedding layer to obtain a plurality of block embedding features.

[0050] This step mainly represents each time-frequency spectrum block as a feature vector. For details, refer to Figure 2 The role of the trained block embedding layer is to represent each time-frequency spectrum block as a feature vector. The block embedding layer can use a linear projection layer, which can be a convolution layer with a large convolution kernel and convolution step. In the specific implementation, this step inputs each 16×16 time-frequency spectrum block into the trained block embedding layer. The block embedding layer can convert each 16×16 time-frequency spectrum block into a 1-dimensional 768-size block embedding feature to obtain multiple block embedding features, such as Figure 2 The block embedding feature E is shown.

[0051] Step S104, inputting the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum blocks into the trained position embedding layer, and obtaining a position embedding feature sequence output by the position embedding layer.

[0052] Specifically, the self-attention model in the subsequent steps cannot capture the order information of the input features by itself, and the aforementioned multiple block embedding features are not arranged in chronological order. Therefore, in this step, the main function of the trained position embedding layer is to obtain a feature sequence containing local information and spatial structure information of the time spectrum graph, wherein the local information of the time spectrum graph is already included in each block embedding feature of the aforementioned step, and the spatial structure information needs to be obtained in combination with the segmentation order information of the time spectrum graph block. The position embedding layer in this step can use a convolutional layer to input multiple block embedding features and the segmentation order information of multiple time spectrum graph blocks into the position embedding layer. The position embedding layer combines each block embedding feature and its segmentation order information to obtain multiple position embedding features, and the multiple position embedding features constitute a position embedding feature sequence. Among them, the position embedding features in the position embedding feature sequence include corresponding block embedding features and position features of the time-spectrogram blocks corresponding to the block embedding features in the time-spectrogram. The position features represent the relative position information of the corresponding time-spectrogram blocks in the time-spectrogram. Thus, the position embedding layer captures the spatial structure information of the two-dimensional audio time-spectrogram. Therefore, the position embedding feature sequence sequentially includes the block embedding features of each time-spectrogram block in the time-spectrogram and the position features of each time-spectrogram block in the time-spectrogram, so that the position embedding feature sequence contains the local information and spatial structure information of the time-spectrogram.

[0053] In some embodiments, the step S104 of inputting the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum blocks into the trained position embedding layer specifically includes:

[0054] A plurality of segmentation order identifiers are generated according to the segmentation order of the plurality of time-frequency spectrum blocks, and the plurality of segmentation order identifiers are correspondingly carried in the plurality of block embedding features; and the plurality of block embedding features carrying the plurality of segmentation order identifiers are input into the position embedding layer.

[0055] In this embodiment, a plurality of segmentation order identifiers are generated according to the segmentation order of a plurality of time-frequency spectrum blocks, so that the segmentation order information is represented by the segmentation order identifiers. Figure 2 As shown, the first segmentation can be represented as CLS, the second segmentation can be represented as 1, and so on, the last segmentation can be represented as N, and CLS, 1, ..., N are multiple segmentation order identifiers; after obtaining multiple segmentation order identifiers, the multiple segmentation order identifiers are correspondingly carried in multiple block embedding features to obtain multiple block embedding features E[CLS], E[1], ..., E[N] carrying multiple segmentation order identifiers, and then the multiple block embedding features E[CLS], E[1], ..., E[N] carrying multiple segmentation order identifiers are input into the position embedding layer. Each block embedding feature can carry a segmentation order identifier.

[0056] Furthermore, the position embedding layer is used to obtain multiple position features corresponding to the multiple block embedding features according to the multiple segmentation order identifiers carried in the input multiple block embedding features (the position features can be obtained by representing the feature vectors of the segmentation order identifiers), combine the corresponding block embedding features and position features to obtain position embedding features, and form a position embedding feature sequence according to the multiple position embedding features obtained by combining. Specifically, Figure 2 As shown, the position embedding layer can represent the multiple segmentation order identifiers carried in the multiple block embedding features with feature vectors to obtain multiple position features P[0], P[1], ..., P[N] corresponding to the multiple block embedding features, and then combine the multiple block embedding features E and the position feature P to obtain the position embedding feature, wherein, if the block embedding feature is 1-dimensional in size 768, the position feature is also 1-dimensional in size 768, and then each block embedding feature E can be added to its corresponding position feature P to obtain multiple position embedding features (E+P), so that the multiple position embedding features sequentially form a position embedding feature sequence.

[0057] Step S105, input the position embedding feature sequence into the encoder of the trained self-attention model to obtain the feature representation of the time-frequency spectrum output by the encoder.

[0058] This step mainly generates the overall feature representation of the spectrum graph based on the acquired position embedding feature sequence. Figure 2 As shown, specifically, in this step, the position embedding feature sequence is converted into the feature representation of the time-frequency spectrum graph through the encoder of the trained self-attention model. The encoder of the self-attention model can adopt the encoder of the Transformer model, that is, in specific applications, the position embedding feature sequence can be converted into the feature representation of the time-frequency spectrum graph through the encoder of the Transformer model. The feature representation can be a 1-dimensional feature vector corresponding to the time-frequency spectrum graph, thereby making full use of the Transformer's ability to express global audio features and achieving sensitive content recognition performance with higher robustness.

[0059] Step S106: Based on the feature representation of the time-frequency spectrum graph, identify whether the audio to be identified contains sensitive content.

[0060] This step mainly determines whether the audio to be identified contains sensitive content based on the overall feature representation of the time-spectrogram. The final judgment result may include that the audio to be identified is audio containing sensitive content or audio not containing sensitive content, and may further be audio containing a certain type of sensitive content, etc. In a specific implementation, the overall feature representation of the time-spectrogram may be a 1-dimensional feature vector. In this step, the 1-dimensional feature vector may be mapped into a sensitive content classification label through a linear layer of a Sigmoid activation function. The sensitive content classification label (the specific form of the label may be 0 or 1) may indicate whether the audio to be identified is audio containing sensitive content.

[0061] The above-mentioned audio sensitive content recognition method extracts the local feature information and spatial structure information of the time spectrum graph through self-learning of the block embedding layer and the position embedding layer to avoid information loss caused by artificial feature selection, and then uses the encoder of the self-attention model to learn the global features of the time spectrum graph to obtain the feature representation of the time spectrum graph and finally identifies whether the audio to be identified is audio containing sensitive content based on the feature representation, thereby improving the accuracy of audio sensitive content recognition. This solution can be applied to the recognition of audio sensitive content in the massive audio and video data generated by Internet products, greatly reducing the labor cost of auditing, and improving recognition efficiency while improving recognition accuracy.

[0062] In some embodiments, the step S106 of identifying whether the audio to be identified contains sensitive content based on the feature representation of the time-spectrogram specifically includes:

[0063] The feature representation of the time-frequency spectrum graph is input into a trained audio sensitive content classifier to obtain a sensitive content classification label output by the audio sensitive content classifier; based on the sensitive content classification label, it is determined whether the audio to be identified contains sensitive content.

[0064] like Figure 2 As shown, this embodiment can be combined with an audio sensitive content classifier to classify and identify audio sensitive content based on the encoder of the block embedding layer, the position embedding layer and the self-attention model. Among them, the audio sensitive content classifier can use a linear layer of a Sigmoid activation function to map the feature representation of the time-frequency spectrum to a sensitive content classification label. The specific form of the sensitive content classification label can be 0 and 1, which respectively represent the inclusion of sensitive content and the exclusion of sensitive content. Its specific form can also be more numerical values ​​to represent the specific type of sensitive content, such as containing sensitive content A, sensitive content B, etc., so that according to the sensitive content classification label output by the audio sensitive content classifier, it can be determined whether the audio to be identified is audio containing sensitive content and what types of sensitive content it contains. Enriching the types of audio sensitive content recognition also facilitates further expansion of the types of audio sensitive content recognition.

[0065] Further, such as Figure 3As shown, in some embodiments, the following steps may also be included:

[0066] Step S301, obtaining a pre-trained audio classification model.

[0067] In this step, the pre-trained audio classification model includes a block embedding layer, a position embedding layer, and an encoder of a self-attention model. The block embedding layer, position embedding layer, and encoder of a self-attention model included in the audio classification model have the same processing flow for audio as the block embedding layer, position embedding layer, encoder of a self-attention model, and audio sensitive content classifier in the aforementioned embodiment. The difference is that the audio classification pre-training model is not a model for sensitive content recognition, but a recognition model for a larger audio classification, such as recognizing whether it contains male voices, female voices, certain types of musical instruments, and other audio classifications. The audio classification pre-training model can be trained using the AudioSet data set.

[0068] Step S302, using the pre-trained block embedding layer, position embedding layer, and encoder of the self-attention model as the encoder of the embedding layer, position embedding layer, and self-attention model to be trained.

[0069] This step is to transfer the encoders of the pre-trained block embedding layer, position embedding layer, and self-attention model in the pre-trained audio classification model as the encoders of the embedding layer, position embedding layer, and self-attention model to be trained. This is because recognition based on the self-attention model requires more training data than CNN, while the amount of training data for audio-sensitive content recognition is not large, and the audio classification pre-training model can be trained using the AudioSet dataset. Therefore, a pre-training mechanism is introduced to train a large audio classification pre-training model based on the AudioSet dataset, so that the encoders of the pre-trained block embedding layer, position embedding layer, and self-attention model in the audio classification pre-training model are used as the encoders of the embedding layer, position embedding layer, and self-attention model to be trained by using transfer learning. On this basis, the training data of audio-sensitive content recognition is used for training to reduce the probability of overfitting and improve universality.

[0070] Step S303: obtaining an audio sample and obtaining a sensitive content classification label corresponding to the audio sample.

[0071] In this step, the number of audio samples can be multiple, including audio samples containing sensitive content and audio samples not containing sensitive content. The sensitive content classification labels corresponding to the audio samples may include containing sensitive content and not containing sensitive content, and may also include what type of sensitive content is contained, etc.

[0072] Step S304: extracting the time-spectrogram of the audio sample as a time-spectrogram sample.

[0073] The steps S303 and S304 can be summarized as obtaining a time-spectrogram sample of an audio sample and obtaining a sensitive content classification label corresponding to the audio sample.

[0074] Step S305, using the spectrogram samples and sensitive content classification labels to train the embedding layer, position embedding layer, encoder of the self-attention model and audio sensitive content classifier of the audio classification model.

[0075] Steps S304 and S305 mainly extract the time-frequency spectrum of the audio sample as the time-frequency spectrum sample, and uniformly train the migrated embedding layer to be trained, the position embedding layer, the encoder of the self-attention model, and the audio sensitive content classifier to be trained according to the time-frequency spectrum sample and the corresponding sensitive content classification label. During the training process, the audio sensitive content classifier outputs a sensitive content classification prediction label, and the sensitive content classification prediction label is compared with the sensitive content classification label. According to the comparison result, the parameters in the embedding layer to be trained, the position embedding layer, the encoder of the self-attention model, and the audio sensitive content classifier to be trained are adjusted until the comparison result shows that the error between the sensitive content classification prediction label and the sensitive content classification label is lower than the set error threshold, and the trained embedding layer, position embedding layer, encoder of the self-attention model, and audio sensitive content classifier are obtained.

[0076] The above-mentioned embodiment can avoid the probability of overfitting to improve universality through the pre-training mechanism and transfer learning when the amount of training data of audio-sensitive content is insufficient, and the training process has the advantage of fast convergence.

[0077] Further, in some embodiments, obtaining the sensitive content classification label corresponding to the audio sample in the above step S303 specifically includes:

[0078] A first label of the audio sample is obtained according to whether the audio sample contains sensitive content; a second label of the audio sample is obtained according to the proportion of sensitive content in the audio sample; and the first label and the second label are used as sensitive content classification labels corresponding to the audio sample.

[0079] This embodiment mainly improves the robustness of audio sensitive content recognition at the label level. Specifically, since each audio has a certain period of audio content, which is sometimes a mixture of sensitive content and non-sensitive content, this embodiment further adds a second label together with the first label as a sensitive content classification label based on whether the audio sample contains sensitive content to obtain the first label of the audio sample (i.e., indicating whether it contains sensitive content). Among them, the second label is obtained based on the proportion of sensitive content in the audio sample, and the proportion of sensitive content can be expressed as the ratio of sensitive content to non-sensitive content in the audio. This ratio is used as the second label, and then the first label and the second label are used together as the sensitive content classification label corresponding to the audio sample. This embodiment can improve the problem of fuzzy sample prediction errors based on the sensitive content classification label including the first label and the second label, and improve the robustness of audio sensitive content recognition.

[0080] Further, in some embodiments, the step S305 uses the spectrogram samples and sensitive content classification labels to train the embedding layer, position embedding layer, encoder of the self-attention model, and audio sensitive content classifier of the audio classification model, specifically including:

[0081] The time-spectrogram samples are masked in time dimension and / or frequency dimension to obtain first interference samples of the time-spectrogram samples; the time-spectrogram samples are mixed in time dimension and / or frequency dimension to obtain second interference samples of the time-spectrogram samples; the embedding layer, position embedding layer, encoder of the self-attention model and audio sensitive content classifier of the audio classification model are trained using the time-spectrogram samples, the first interference samples, the second interference samples and sensitive content classification labels.

[0082] This embodiment mainly improves the robustness of audio-sensitive content recognition at the sample level. The recognition of audio-sensitive content has the characteristics of small sample size and uneven sample distribution. This embodiment can form interference samples of the time-frequency spectrum sample through masking, mixing and other processing to enrich the training samples and improve the robustness of audio-sensitive content recognition. Specifically, the mask processing can include mask processing in the time dimension and the frequency dimension, such as Figure 2 As shown, the masking process in the time dimension may be to set the intensity of all frequencies corresponding to a certain time to 0 or 1, and the masking process in the frequency dimension may be to set the intensity of all times corresponding to a certain frequency to 0 or 1, thereby performing masking process in the time dimension and / or frequency dimension on the time-spectrum graph sample to obtain the first interference sample of the time-spectrum graph sample. For the mixed processing, it may include mixed processing in the time dimension and the frequency dimension, such as Figure 2As shown, the mixing process in the time dimension can be to superimpose or replace the intensity of all frequencies corresponding to a certain period of time with the intensity of all frequencies corresponding to another period of time to achieve a mixing effect, and the mixing process in the frequency dimension can be to superimpose or replace the intensity of all times corresponding to a certain frequency segment with the intensity of all times corresponding to another frequency segment to achieve a mixing effect, thereby the time-spectrogram sample can be mixed in the time dimension and / or frequency dimension to obtain the second interference sample of the time-spectrogram sample. The acquisition of the first interference sample and the second interference sample enriches the training samples, and then the embedding layer to be trained, the position embedding layer, the encoder of the self-attention model and the audio sensitive content classifier to be trained are trained using the time-spectrogram sample, the first interference sample, the second interference sample and the sensitive content classification label to improve their anti-interference and robustness to audio content recognition.

[0083] In one embodiment, a method for identifying audio-sensitive content is also provided. Figure 4 As shown, the method comprises the following steps:

[0084] Step S401, obtaining original audio.

[0085] Step S402: input the original audio into the first audio sensitive content recognition model to obtain a plurality of audio segments containing sensitive content in the original audio output by the first audio sensitive content recognition model.

[0086] Step S403: Input multiple audio clips as audio to be recognized into the second audio sensitive content recognition model to obtain a sensitive content recognition result for each audio clip output by the second audio sensitive content recognition model.

[0087] Step S404: Determine whether the original audio is sensitive audio based on the sensitive content identification result of each audio segment.

[0088] This embodiment mainly adopts the method of serially identifying the first audio sensitive content recognition model and the second audio sensitive content recognition model to determine whether the original audio is sensitive audio. Among them, the first audio sensitive content recognition model can adopt the CRNN model, and the second audio sensitive content recognition model identifies whether the audio to be identified is audio containing sensitive content according to the audio sensitive content recognition method provided in the above embodiments of this application. The first audio sensitive content recognition model is used to locate the audio segment containing sensitive content in the audio, and the second audio sensitive content recognition model is used to identify the input audio to determine whether the audio contains sensitive content.

[0089] refer to Figure 5Specifically, the original audio can be first input into the first audio sensitive content recognition model, which performs preliminary sensitive content recognition and locates and outputs multiple audio segments containing sensitive content in the original audio. Each audio segment is identified by the first audio sensitive content recognition model as suspected of containing sensitive content. Then, each audio segment is input as an audio to be identified into the second audio sensitive content recognition model for secondary recognition and confirmation. The second audio sensitive content recognition model identifies whether each audio segment is audio containing sensitive content according to the audio sensitive content recognition method provided in the above embodiments of the present application, and obtains a sensitive content recognition result for each audio segment. Finally, it is determined whether the original audio is sensitive audio based on the sensitive content recognition result of each audio segment. In a specific application, if the second audio sensitive content recognition model determines that there is at least one audio segment containing sensitive content among the multiple audio segments (such as when the probability that the second audio sensitive content recognition model identifies that the audio segment contains sensitive content is higher than a set probability threshold), the original audio can be determined as sensitive audio.

[0090] The solution of this embodiment provides a coarse-first-then-fine audio sensitive content recognition method, which first roughly locates the suspected audio segments contained in the entire original audio based on the first audio sensitive content recognition model, thereby having a higher recall rate for the suspected sensitive audio in this link, and then uses the second audio sensitive content recognition model to perform more refined secondary recognition and confirmation, thereby improving the accuracy of audio content recognition in this link, so as to enhance the overall audio sensitive content recognition performance, especially when there are multiple sensitive content segments in long audio, achieving better recognition performance.

[0091] In another embodiment, a method for identifying audio-sensitive content is also provided. Figure 6 As shown, the method includes:

[0092] Step S601, obtaining original audio.

[0093] Step S602: input the original audio as the audio to be recognized into the second audio sensitive content recognition model, and obtain the sensitive content recognition result for the original audio output by the second audio sensitive content recognition model.

[0094] Step S603: When the sensitive content recognition result output by the second audio sensitive content recognition model indicates that the original audio is audio containing sensitive content, the original audio is input into the first audio sensitive content recognition model to obtain the sensitive content recognition result for the original audio output by the first audio sensitive content recognition model.

[0095] Step S604: Determine whether the original audio is sensitive audio based on the sensitive content recognition result output by the first audio sensitive content recognition model.

[0096] In this embodiment, the difference from the previous embodiment is that this embodiment performs primary screening through the second audio sensitive content recognition model and performs secondary recognition through the first audio sensitive content recognition model. Specifically, the original audio is first input into the second audio sensitive content recognition model as the audio to be recognized, and the sensitive content recognition result for the original audio output by the second audio sensitive content recognition model is obtained. The sensitive content recognition result can be whether the original audio is audio containing sensitive content. If so, the original audio is further input into the first audio sensitive content recognition model. The first audio sensitive content recognition model can further determine whether the original audio contains sensitive content and output an audio segment containing sensitive content. These can all be used as the sensitive content recognition results output by the first audio sensitive content recognition model. Therefore, when the first audio sensitive content recognition model recognizes and outputs at least one audio segment, the original audio can be determined to be sensitive audio, so as to achieve the effect of accurately identifying and quickly locating the audio segment containing sensitive content in the original audio.

[0097] In yet another embodiment, a method for identifying audio-sensitive content is also provided. Figure 7 As shown, the method includes:

[0098] Step S701, obtaining original audio.

[0099] Step S702: input the original audio into the first audio sensitive content recognition model to obtain the sensitive content recognition result for the original audio output by the first audio sensitive content recognition model.

[0100] Step S703: input the original audio as the audio to be recognized into the second audio sensitive content recognition model, and obtain the sensitive content recognition result for the original audio output by the second audio sensitive content recognition model.

[0101] Step S704: Determine whether the original audio is sensitive audio based on the sensitive content recognition result output by the first audio sensitive content recognition model and the sensitive content recognition result output by the second audio sensitive content recognition model.

[0102] In this embodiment, different from the above-mentioned embodiment, this embodiment recognizes sensitive content of the original audio by connecting the first audio sensitive content recognition model and the second audio sensitive content recognition model in parallel. Specifically, after obtaining the original audio, on the one hand, the original audio is input into the first audio sensitive content recognition model to obtain the sensitive content recognition result output by the first audio sensitive content recognition model, and on the other hand, the original audio is input into the second audio sensitive content recognition model to obtain the sensitive content recognition result output by the second audio sensitive content recognition model, and then the two results are combined to determine whether the original audio is sensitive audio. Exemplarily, if the recognition results of the two models are that the original audio is not sensitive audio, then the original audio is determined to be not sensitive audio; if the recognition results of the two models are that the original audio is sensitive audio, then the original audio is determined to be sensitive audio; if the recognition results of the two models are inconsistent, such as the recognition result of one model is that the probability that the original audio belongs to sensitive audio is P1, and the probability given by the other model is P2, then the two probabilities can be weighted according to the weights assigned to the two models to obtain a comprehensive probability P, and the original audio is judged to be sensitive audio according to the comprehensive probability. For example, if the comprehensive probability P is greater than or equal to the set probability threshold, the original audio is judged to be sensitive audio, otherwise it is judged not to be sensitive audio. The solution of this embodiment can assign corresponding weights to the two models in different audio recognition scenarios (such as long audio and video, short audio and video scenarios) to reflect the recognition advantages of different models in different audio recognition scenarios and improve recognition accuracy.

[0103] The method for identifying audio-sensitive content provided in this application can be applied to the review of audio clips of karaoke products and the like, and can also be applied to the review of film and television works or audio and video program content. For example, audio clips (short audio and video, live broadcast real-time audio) uploaded by users can be used to identify audio-sensitive content, and the identification results can be displayed on the terminal devices of relevant personnel, such as Figure 8 As shown, the specific information displayed may include the identified audio clip and the identification result and its confidence (i.e., the probability of belonging to sensitive content A) for further manual review and other processing. This application can realize intelligent recognition of audio sensitive content in massive audio and video data, greatly reducing the labor cost of audio sensitive content review and greatly improving the efficiency and timeliness of review.

[0104] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0105] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a communication interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as audio to be identified. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for identifying audio-sensitive content is implemented.

[0106] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0108] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0109] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in each embodiment provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto.

[0111] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0112] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying audio-sensitive content, characterized in that: The method comprises: Obtaining a time-spectrogram of the audio to be recognized, and dividing the time-spectrogram into a plurality of time-spectrogram blocks; Obtaining block embedding features corresponding to each of the plurality of time-frequency spectrum blocks through a trained block embedding layer to obtain a plurality of block embedding features; Input the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum blocks into the trained position embedding layer to obtain a position embedding feature sequence output by the position embedding layer; the position embedding features in the position embedding feature sequence include block embedding features and position features of the time-frequency spectrum blocks corresponding to the block embedding features in the time-frequency spectrum; Inputting the position embedding feature sequence into an encoder of a trained self-attention model to obtain a feature representation of the time-spectrogram output by the encoder; Based on the feature representation of the time-frequency spectrum graph, it is identified whether the audio to be identified contains sensitive content.

2. The method according to claim 1, characterized in that The step of dividing the time-frequency spectrum graph into a plurality of time-frequency spectrum graph blocks comprises: The time-frequency spectrum diagram is segmented according to a preset segmentation order and a preset number of overlapping dimensions to obtain the multiple time-frequency spectrum diagram blocks; the preset number of overlapping dimensions includes the number of overlapping dimensions in the time dimension and / or the frequency dimension.

3. The method according to claim 1, characterized in that The step of inputting the plurality of block embedding features and the segmentation order information of the plurality of time-frequency spectrum blocks into the trained position embedding layer comprises: Generating a plurality of segmentation order identifiers according to the segmentation order of the plurality of time-frequency spectrum blocks, and carrying the plurality of segmentation order identifiers in the plurality of block embedding features accordingly; The plurality of block embedding features carrying the plurality of segmentation order identifiers are input into the position embedding layer.

4. The method according to claim 3, characterized in that The position embedding layer is used to obtain multiple position features corresponding to the multiple block embedding features according to the multiple segmentation order identifiers carried in the multiple block embedding features, combine the corresponding block embedding features and position features to obtain position embedding features, and form the position embedding feature sequence according to the multiple position embedding features obtained by combination.

5. The method according to any one of claims 1 to 4, characterized in that: The identifying whether the audio to be identified contains sensitive content based on the feature representation of the time-frequency spectrum graph includes: Inputting the feature representation of the time-frequency spectrum into a trained audio sensitive content classifier to obtain a sensitive content classification label output by the audio sensitive content classifier; Determine whether the audio to be identified contains sensitive content based on the sensitive content classification label.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining a pre-trained audio classification model, wherein the audio classification model includes a block embedding layer, a position embedding layer, and an encoder of a self-attention model; Obtaining a time-frequency spectrogram sample of an audio sample and obtaining a sensitive content classification label corresponding to the audio sample; The embedding layer, position embedding layer, encoder of the self-attention model and the audio sensitive content classifier of the audio classification model are trained using the time-spectrogram samples and the sensitive content classification labels.

7. The method according to claim 6, characterized in that The obtaining of the sensitive content classification label corresponding to the audio sample includes: Obtaining a first label of the audio sample according to whether the audio sample contains sensitive content; Obtaining a second label of the audio sample according to a proportion of sensitive content in the audio sample; The first label and the second label are used as sensitive content classification labels corresponding to the audio sample.

8. The method according to claim 6, characterized in that The method of using the spectrogram samples and the sensitive content classification labels to train the embedding layer, the position embedding layer, the encoder of the self-attention model, and the audio sensitive content classifier of the audio classification model includes: Performing mask processing on the time-frequency spectrum sample in a time dimension and / or a frequency dimension to obtain a first interference sample of the time-frequency spectrum sample; Performing mixed processing of the time dimension and / or frequency dimension on the time-spectrogram sample to obtain a second interference sample of the time-spectrogram sample; The embedding layer, position embedding layer, encoder of the self-attention model and the audio sensitive content classifier of the audio classification model are trained using the time-spectrogram samples, the first interference samples, the second interference samples and the sensitive content classification labels.

9. A method for identifying audio-sensitive content, characterized in that: include: Get the original audio; Input the original audio into a first audio sensitive content recognition model to obtain a plurality of audio segments containing sensitive content in the original audio output by the first audio sensitive content recognition model; input the plurality of audio segments as audio to be recognized into a second audio sensitive content recognition model to obtain a sensitive content recognition result for each audio segment output by the second audio sensitive content recognition model; determine whether the original audio is sensitive audio according to the sensitive content recognition result of each audio segment; or, Inputting the original audio as the audio to be recognized into a second audio sensitive content recognition model, and obtaining a sensitive content recognition result for the original audio output by the second audio sensitive content recognition model; When the sensitive content recognition result output by the second audio sensitive content recognition model indicates that the original audio is audio containing sensitive content, the original audio is input into the first audio sensitive content recognition model to obtain the sensitive content recognition result output by the first audio sensitive content recognition model for the original audio; and according to the sensitive content recognition result output by the first audio sensitive content recognition model, determine whether the original audio is sensitive audio; or, Input the original audio into a first audio sensitive content recognition model to obtain a sensitive content recognition result for the original audio output by the first audio sensitive content recognition model; input the original audio as the audio to be recognized into a second audio sensitive content recognition model to obtain a sensitive content recognition result for the original audio output by the second audio sensitive content recognition model; determine whether the original audio is sensitive audio according to the sensitive content recognition result output by the first audio sensitive content recognition model and the sensitive content recognition result output by the second audio sensitive content recognition model; The second audio sensitive content recognition model is used to identify whether the audio to be recognized contains sensitive content according to the method described in any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images