An audio detection method and apparatus
Through the combination of two rounds of vocal effectiveness detection and violation detection, the problem of low audio detection accuracy in the existing technology is solved, the detection accuracy and efficiency are improved, and the misjudgment rate is reduced.
Patent Information
- Application Number
- CN202211020912.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-08-24
AI Technical Summary
Due to the single detection rules of the existing audio detection scheme, the non-violated audio is misjudged as a violation, and the detection accuracy is low.
Using two-round vocal effectiveness detection method, firstly, determine whether the audio data contains vocals through the first vocal effectiveness detection. If included, conduct a second vocal effectiveness detection to accurately judge. If not included, directly determine that there is no violation. , if there is any illegal sound, the violation will be tested.
It improves the accuracy of audio detection, shortens the detection process, reduces the operating costs of the server, reduces misjudgment, and enhances the ability to identify illegal content.
Smart Images

Figure CN115394318B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an audio detection method and device. Background Art
[0002] As various pan-entertainment social networking or live broadcasting fields are promoted more and more widely, more and more users are using pan-entertainment social networking or live broadcasting. Tens of thousands to hundreds of thousands of hours of audio content are generated on the Internet every day, but there will always be people who maliciously spread illegal or illegal content. Therefore, in order to ensure the security and legality of the network, it is necessary to detect whether the audio data generated by users is illegal in real time.
[0003] The existing audio detection scheme for determining whether audio is in violation of the rules may misjudge audio that is not in violation of the rules as in violation of the rules due to the single detection rule, resulting in low detection accuracy. Summary of the invention
[0004] In view of this, the present application provides an audio detection method and device for solving the problem that the existing audio detection scheme for determining whether audio is in violation of regulations may misjudge non-violation audio as in violation of regulations due to the single detection rule, resulting in low detection accuracy.
[0005] To achieve the above objectives, the proposed solution is as follows:
[0006] In a first aspect, an audio detection method includes:
[0007] Get audio data generated by the user;
[0008] Performing a first human voice validity detection on the audio data to obtain a first human voice validity ratio of the audio data;
[0009] If the first human voice validity ratio is less than a preset first ratio threshold, it is determined that the audio data is not in violation; if the first human voice validity ratio is not less than the preset first ratio threshold, a second human voice validity detection is performed on the audio data to obtain a second human voice validity score of the audio data;
[0010] If the second human voice validity score is less than a preset second validity threshold, determining that the audio data does not violate the rules;
[0011] If the second human voice validity score is not less than a preset second validity threshold, determining whether the audio data contains an illegal sound;
[0012] If the audio data contains illegal sounds, it is determined that the audio data is illegal.
[0013] Preferably, performing the first human voice validity detection on the audio data to obtain the first human voice validity ratio of the audio data includes:
[0014] Converting the audio data into a spectrogram;
[0015] The first human voice validity detection model is used to process the spectrogram using a pre-trained first human voice validity detection model to obtain a first human voice validity score for each frame of the audio data; the first human voice validity detection model is trained using the spectrogram of the audio data sample as a training sample and the first human voice validity score of each frame of the audio data sample as a sample label;
[0016] Based on the first human voice effectiveness score of each frame of the audio data, a first human voice effectiveness ratio of the audio data is obtained.
[0017] Preferably, obtaining the first human voice effectiveness ratio of the audio data based on the first human voice effectiveness score of each frame of the audio data includes:
[0018] For each frame of the audio data, if the first human voice validity score of the frame of audio data is less than a preset first validity threshold, it is determined that the frame of audio data does not contain a human voice; if the first human voice validity score of the frame of audio data is not less than the first validity threshold, it is determined that the frame of audio data contains a human voice;
[0019] Obtaining the total number of frames of the audio data, and counting the number of frames of the audio data containing human voice;
[0020] Based on the total number of frames of the audio data and the number of frames of the audio data containing human voice, a first human voice effectiveness ratio of the audio data is determined.
[0021] Preferably, performing a second human voice validity detection on the audio data to obtain a second human voice validity score of the audio data includes:
[0022] Converting the audio data into a spectrogram;
[0023] The spectrogram is processed using a pre-trained second voice effectiveness detection model to obtain a second voice effectiveness score of the audio data; the second voice effectiveness detection model is trained using the spectrogram of the audio data sample as a training sample and the second voice effectiveness score of the audio data sample as a sample label.
[0024] Preferably, the using a pre-trained second human voice validity detection model to process the spectrogram to obtain a second human voice validity score of the audio data includes:
[0025] Using the convolution module of the second human voice validity detection model, extracting spatial features from the spectrogram to obtain spatial feature information of the audio data;
[0026] Using the loop module of the second human voice validity detection model, extracting time domain features from the spatial feature information to obtain time domain feature information of the audio data;
[0027] Using the fully connected module of the second human voice effectiveness detection model, dimensionally adjust the time domain feature information to obtain a second human voice effectiveness sub-score for each frame of the audio data;
[0028] The second vocal effectiveness sub-score of each frame is pooled using the pooling module of the second vocal effectiveness detection model to obtain the second vocal effectiveness score of the audio data.
[0029] Preferably, the determining whether the audio data contains illegal sounds comprises:
[0030] Processing the audio data using a preset violation detection model to obtain a first violation score corresponding to the audio data; the violation detection model is trained using the audio data sample as a training sample and the first violation score of the audio data as a sample label;
[0031] If the first violation score of the audio data is not less than a preset first violation threshold, it is determined that the audio data contains an illegal sound.
[0032] Preferably, the method further comprises:
[0033] If the first violation score of the audio data is less than the first violation threshold, determining text data corresponding to the audio data based on the audio data, and judging whether the text data contains a violation text;
[0034] If the text data does not contain any illegal text, then determining that the audio data does not violate the rules;
[0035] If the text data contains illegal text, the audio data is determined to be illegal.
[0036] Preferably, the determining whether the text data contains illegal text includes:
[0037] Perform keyword matching on the text data based on a preset illegal keyword word library, and if the keyword matching is successful, determine that the text data contains illegal text;
[0038] If the keyword matching is unsuccessful, semantic analysis is performed on the text data to obtain a second violation score of the audio data;
[0039] If the second violation score is less than a preset second violation threshold, determining that the text data does not contain a violation text;
[0040] If the second violation score is not less than the second violation score, it is determined that the text data contains violation text.
[0041] In a second aspect, an audio detection device includes:
[0042] An acquisition module, used to acquire audio data generated by a user;
[0043] A first validity detection module, configured to perform a first human voice validity detection on the audio data to obtain a first human voice validity ratio of the audio data;
[0044] A second validity detection module is used to determine that the audio data is not in violation of the rules when the first human voice validity ratio is less than a preset first ratio threshold; and to perform a second human voice validity detection on the audio data to obtain a second human voice validity score of the audio data when the first human voice validity ratio is not less than the preset first ratio threshold;
[0045] A first violation determination module, configured to determine that the audio data does not violate the rule when the second voice validity score is less than a preset second validity threshold;
[0046] A judgment module, configured to judge whether the audio data contains an illegal sound when the second human voice validity score is not less than a preset second validity threshold;
[0047] The second violation determination module is used to determine that the audio data violates the rules when the audio contains illegal sounds.
[0048] Preferably, the first validity detection module includes:
[0049] A conversion module, used for converting the audio data into a spectrogram;
[0050] A first human voice effectiveness score obtaining module is used to process the spectrogram using a pre-trained first human voice effectiveness detection model to obtain a first human voice effectiveness score for each frame of the audio data; the first human voice effectiveness detection model is trained using the spectrogram of the audio data sample as a training sample and the first human voice effectiveness score of each frame of the audio data sample as a sample label;
[0051] The first human voice effectiveness ratio determination module is used to obtain the first human voice effectiveness ratio of the audio data based on the first human voice effectiveness score of each frame of the audio data.
[0052] It can be seen from the above technical scheme that the present application obtains the audio data generated by the user and performs a first human voice validity detection on the audio data to obtain the first human voice validity ratio of the audio data; if the first human voice validity ratio is less than the preset first ratio threshold, it is determined that the audio data is not in violation of the rules; if the first human voice validity ratio is not less than the preset first ratio threshold, the second human voice validity detection is performed on the audio data to obtain the second human voice validity score of the audio data; if the second human voice validity score is less than the preset second validity threshold, it is determined that the audio data is not in violation of the rules; if the second human voice validity score is not less than the preset second validity threshold, it is determined whether the audio data contains illegal sounds; if the audio contains illegal sounds, it is determined that the audio data is in violation of the rules. This scheme performs a first human voice validity detection on the audio data generated by the user, and can obtain the first human voice validity ratio of the audio data, and compare the ratio with the preset first ratio threshold, so that it can directly determine whether the audio data contains human voices. If it does not contain human voices, it is not necessary to enter the subsequent judgment, which can shorten the detection process of the audio data. If it does, the second human voice validity test will be entered to make an accurate judgment on whether it contains human voice. If after two rounds of detection, the audio data is determined to contain no human voice, then it is determined that the audio data does not contain human voice, that is, it does not violate the rules. If the second round determines that it contains human voice, the audio data will be detected for illegal sounds. If the audio data contains illegal sounds, then the audio data is determined to be illegal, thereby improving the accuracy of the detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0054] Figure 1 An optional flow chart of an audio detection method provided in an embodiment of the present application;
[0055] Figure 2 An optional flow chart of another audio detection method provided in an embodiment of the present application;
[0056] Figure 3 A schematic diagram of the structure of an audio detection device provided in an embodiment of the present application;
[0057] Figure 4 A schematic diagram of the structure of an audio detection device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0059] As the promotion of various pan-entertainment social networking or live broadcasting fields becomes more and more widespread, more and more users are using pan-entertainment social networking or live broadcasting. Tens of thousands to hundreds of thousands of hours of audio content are generated on the Internet every day, but there will always be people who maliciously spread illegal or illegal content. Therefore, in order to ensure the security and legality of the network, it is necessary to detect whether the audio data generated by users is illegal in real time. If illegal audio is detected, the illegal information needs to be deleted. If it involves serious circumstances, the user who generated the illegal audio will be banned and other measures will be taken to maintain the network environment.
[0060] For the above problems, if manual verification is used, the manpower cost is huge and tedious. The existing audio detection method has benefited from the development of deep learning. Using deep learning models to review audio is a common solution in the industry. However, the existing audio detection scheme for whether the audio is in violation of the rules will misjudge the audio that is not in violation as in violation due to the single detection rules, resulting in low detection accuracy.
[0061] Based on the above defects, an embodiment of the present invention provides an audio detection method, which can be applied to various live broadcast systems or voice chat systems, and can also be applied to various computer terminals or smart terminals. The execution subject can be a processor or server of the computer terminal or smart terminal. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:
[0062] S1: Obtain audio data generated by the user.
[0063] In the embodiment of the present invention, when acquiring audio data generated by a user, the audio data of one of the users within a certain period of time may be acquired, or each frame of audio data generated by one user may be acquired in real time.
[0064] S2: Perform a first voice validity detection on the audio data to obtain a first voice validity ratio of the audio data.
[0065] The first human voice validity detection is a method of roughly detecting whether audio contains human voice. According to the first human voice validity detection method, the proportion of audio containing human voice can be roughly detected as the first human voice validity proportion of the audio data generated by the user.
[0066] S3: If the first voice effectiveness ratio is less than the preset first ratio threshold, it is determined that the audio data is not in violation of the rules; if the first voice effectiveness ratio is not less than the preset first ratio threshold, a second voice effectiveness detection is performed on the audio data to obtain the second voice effectiveness score of the audio data.
[0067] In the present application, a first ratio threshold can be preset to indicate the minimum ratio of human voice in a segment of audio data. If the first human voice validity ratio of the audio data generated by the user is less than the first ratio threshold, it can be determined that the probability that the audio data does not contain human voice is low, and therefore the probability that the audio data violates the rules is low, and it can be determined that the audio data does not violate the rules, so there is no need to enter the subsequent detection process, which can shorten the audio data detection process.
[0068] It is understandable that the second human voice validity detection can detect whether the audio data contains human voice in a relatively precise manner. If the first human voice validity ratio of the audio data generated by the user is not less than the preset first ratio threshold, it means that the probability of containing human voice in the audio data is relatively high, that is, it can be considered that the audio data may be in violation of regulations, so the second human voice validity detection is performed on the audio data to obtain the second human voice validity score of the audio data.
[0069] S4: If the second voice validity score is less than a preset second validity threshold, it is determined that the audio data does not violate the rules.
[0070] Based on the above step S3, the second human voice effectiveness score of the audio data generated by the user is obtained, and the second human voice effectiveness score can be compared with the preset second effectiveness threshold. If the second human voice effectiveness score is less than the preset second effectiveness threshold, it can be explained that the probability of detecting that the audio data contains human voice using a more accurate second human voice effectiveness detection method is not that high. Therefore, the audio data can be judged as not containing human voice, and therefore the audio data does not violate the regulations.
[0071] S5: If the second voice validity score is not less than a preset second validity threshold, determine whether the audio data contains any illegal sounds.
[0072] In this step, if the second human voice validity score is less than the preset second validity threshold, it can be said that the probability of the audio data containing human voice is relatively high, and then it can be determined whether the audio data contains illegal sounds.
[0073] Optionally, a method for determining whether the audio data contains illegal sounds includes, but is not limited to: comparing a preset illegal sound set with the audio data; if the comparison is successful, it means that the audio data contains illegal sounds.
[0074] S6: If the audio contains illegal sounds, it is determined that the audio data is illegal.
[0075] It can be seen from the above technical scheme that the present application obtains the audio data generated by the user and performs a first human voice validity detection on the audio data to obtain the first human voice validity ratio of the audio data; if the first human voice validity ratio is less than the preset first ratio threshold, it is determined that the audio data is not in violation of the rules; if the first human voice validity ratio is not less than the preset first ratio threshold, the second human voice validity detection is performed on the audio data to obtain the second human voice validity score of the audio data; if the second human voice validity score is less than the preset second validity threshold, it is determined that the audio data is not in violation of the rules; if the second human voice validity score is not less than the preset second validity threshold, it is determined whether the audio data contains illegal sounds; if the audio contains illegal sounds, it is determined that the audio data is in violation of the rules. This scheme performs a first human voice validity detection on the audio data generated by the user, and can obtain the first human voice validity ratio of the audio data, and compare the ratio with the preset first ratio threshold, so that it can directly determine whether the audio data contains human voices. If it does not contain human voices, it is not necessary to enter the subsequent judgment, which can shorten the detection process of the audio data. If it does, the second human voice validity test will be entered to make an accurate judgment on whether it contains human voice. If after two rounds of detection, the audio data is determined to contain no human voice, then it is determined that the audio data does not contain human voice, that is, it does not violate the rules. If the second round determines that it contains human voice, the audio data will be detected for illegal sounds. If the audio data contains illegal sounds, then the audio data is determined to be illegal, thereby improving the accuracy of the detection.
[0076] It is understandable that, in one example, the first human voice validity detection process can be set on the user side, such as various user-side devices such as mobile phone user terminals or user computer terminals, and the second human voice validity detection process can be set on the server side. If multiple users generate 100,000 hours of audio a day, then after the first human voice validity detection, 60,000 hours of audio that do not contain human voices are excluded. Therefore, only 40,000 hours of audio enter the second human voice validity detection process. In the second human voice validity detection process, only these 40,000 hours of audio need to be detected, thereby effectively reducing the operating costs of the server and reducing pressure. Next, when the second human voice validity detection process accurately detects these 40,000 hours of audio, it will screen out the audio that is accurately detected as not containing human voices, thereby further reducing the subsequent detection process and minimizing the operating costs of the server.
[0077] In the method provided in the embodiment of the present invention, the specific process of performing the first human voice validity detection on the audio data and obtaining the first human voice validity ratio of the audio data may include:
[0078] S21: Convert the audio data into a spectrogram.
[0079] In this step, the audio data can be decomposed into a DC component and several sinusoidal signals by Fourier transform, and each sinusoidal signal has its own frequency and amplitude. Optionally, the frequency value of the audio data is taken as the horizontal axis, and the amplitude of the audio data is taken as the vertical axis, and the amplitude of several sinusoidal signals of the audio data is combined with the corresponding frequency, so as to obtain the spectrum diagram corresponding to the audio data.
[0080] S22: Use a pre-trained first human voice validity detection model to process the spectrogram to obtain a first human voice validity score for each frame of the audio data; the first human voice validity detection model is trained using the spectrogram of the audio data sample as a training sample and the first human voice validity score of each frame of the audio data sample as a sample label.
[0081] Specifically, in the first human voice validity detection model, feature information can be extracted through a convolutional neural network, wherein a MobileNetV2 lightweight network can be used as a convolutional neural network, which can increase the model running speed and reduce user perception. The model can roughly detect whether the audio data contains human voices, so various audios that do not contain human voices, or audios that only contain background music, etc. can be used as audio data samples to train the first human voice validity detection model. The first human voice validity detection model requires fewer parameters, has a fast calculation speed, and has a high human voice recall, and can quickly and roughly detect whether the audio data contains human voices.
[0082] S23: Obtaining a first voice effectiveness ratio of the audio data based on the first voice effectiveness score of each frame of the audio data.
[0083] Optionally, step S23 may include the following steps:
[0084] S231: For each frame of the audio data, if the first voice validity score of the frame audio data is less than a preset first validity threshold, it is determined that the frame audio data does not contain a human voice; if the first voice validity score of the frame audio data is not less than the first validity threshold, it is determined that the frame audio data contains a human voice.
[0085] The first preset threshold can be set to 50 points, which is not limited in this embodiment. With 50 points as the boundary, for each frame of audio data, if the first human voice validity score of the audio data is less than 50 points, it is determined that the audio data of this frame does not contain human voice; if it is greater than 50 points, it is determined that the audio data of this frame contains human voice.
[0086] However, each frame of audio data contains too little audio information (every 25 ms can be regarded as one frame of audio data), so it is necessary to analyze the human voice contained in the total frame of audio data to determine the effectiveness ratio of the first human voice.
[0087] S232: Obtain the total number of frames of the audio data, and count the number of frames of the audio data containing human voice.
[0088] S233: Determine a first human voice effectiveness ratio of the audio data based on the total number of frames of the audio data and the number of frames of the audio data containing human voice.
[0089] According to the analysis of each frame of the entire audio, after obtaining the total number of frames of audio data and the number of frames containing human voice, the number of frames containing human voice is divided by the total number of frames of audio data to obtain the first human voice effectiveness ratio of the audio data, that is, the probability that the audio data contains human voice.
[0090] The above embodiment describes the process of how to perform a first human voice validity detection on the audio data and obtain the first human voice validity ratio of the audio data in the present application. The following describes in detail the specific process of how to perform a second human voice validity detection on the audio data and obtain the second human voice validity score of the audio data.
[0091] The audio data is converted into a spectrogram, and then the spectrogram is processed using a pre-trained second voice effectiveness detection model to obtain a second voice effectiveness score of the audio data; the second voice effectiveness detection model is trained using the spectrogram of the audio data sample as a training sample and the second voice effectiveness score of the audio data sample as a sample label.
[0092] Specifically, the process may include:
[0093] S31: Utilize the convolution module of the second human voice validity detection model to extract spatial features of the spectrum graph to obtain spatial feature information of the audio data.
[0094] The convolution module includes a convolutional neural network (CNN), which is used to extract spatial features from the spectrogram of audio data. The convolutional neural network can select a VGGISH network that is larger than the MobileNetV2 lightweight network to improve the accuracy of the model.
[0095] S32: Utilize the loop module of the second human voice validity detection model to perform time domain feature extraction on the spatial feature information to obtain the time domain feature information of the audio data.
[0096] The recurrent module may include a recurrent neural network (RNN) for extracting features in the time domain from the spatial feature information obtained in the above step S31.
[0097] S33: Utilize the fully connected module of the second vocal effectiveness detection model to perform dimension adjustment on the time domain feature information to obtain the second vocal effectiveness sub-score of each frame of the audio data.
[0098] The fully connected module includes a fully connected layer (Fully Connected, FC for short), which can be used to dimensionally adjust the time domain feature information obtained in the above steps to obtain the second vocal effectiveness sub-score of each frame of the audio data.
[0099] S34: Utilize the pooling module of the second vocal effectiveness detection model to perform pooling processing on the second vocal effectiveness sub-scores of each frame to obtain the second vocal effectiveness score of the audio data.
[0100] In the above step S33, the second vocal effectiveness sub-score of each frame of the audio data is obtained. The second vocal effectiveness score of the audio data can be obtained by performing pooling processing through the pooling module of the second vocal effectiveness detection model. Among them, the pooling module can be set to maximum pooling, average pooling or linear-softmax. The use of linear-softmax can make the second vocal effectiveness detection model more reasonable when performing reverse conduction during training, and make the trained second vocal effectiveness detection model more accurate. The calculation formula for the second vocal effectiveness score Y using linear-softmax can be expressed as:
[0101]
[0102] Among them, y i Represents the second vocal effectiveness sub-score of the i-th frame of audio data.
[0103] It can be understood that the second voice validity detection model has higher detection accuracy and is more accurate than the first voice validity detection model.
[0104] Preferably, in step S5, the process of determining whether the audio data contains illegal sounds may include:
[0105] The audio data is processed using a preset violation detection model to obtain a first violation score corresponding to the audio data; the violation detection model is trained using the audio data samples as training samples and the first violation score of the audio data as a sample label.
[0106] If the first violation score of the audio data is not less than a preset violation threshold, it is determined that the audio data contains an illegal sound.
[0107] If the first violation score of the audio data is less than the violation threshold, then based on the audio data, text data corresponding to the audio data is determined. Preferably, an automatic speech recognition model (Automatic Speech Recognition, referred to as ASR) can be used to convert the audio data into text data.
[0108] Determine whether the text data contains illegal text; if the text data does not contain illegal text, determine that the audio data does not violate the rules; if the text data contains illegal text, determine that the audio data violates the rules.
[0109] Specifically, the process of determining whether the text data contains illegal text may include:
[0110] Keyword matching is performed on the text data based on a preset illegal keyword vocabulary. If the keyword matching is successful, it is determined that the text data contains illegal text; if the keyword matching is unsuccessful, semantic analysis is performed on the text data to obtain a second violation score of the audio data; if the second violation score is less than a preset second violation threshold, it is determined that the text data does not contain illegal text; if the second violation score is not less than the second violation score, it is determined that the text data contains illegal text.
[0111] In the embodiment provided in the present application, the overall scheme process of an audio detection method can be as follows: Figure 2 shown.
[0112] It is understandable that if it is finally determined that the audio data violates the regulations, the non-violation sound part of the audio data can be deleted, and then the audio data after deleting the violation sound can be sent to the manual review link. If the staff finds that the violation sound involves more serious circumstances, the user who generated the violation sound can be punished according to the specific circumstances to maintain network civilization and security.
[0113] and Figure 1 Corresponding to the method described above, the embodiment of the present invention further provides an audio detection device for detecting Figure 1 In the specific implementation of the method, the audio detection device provided in the embodiment of the present invention can be used in a computer terminal or various mobile devices, combined with Figure 3 , the audio detection device is introduced, such as Figure 3 As shown, the device may include:
[0114] The acquisition module 10 is used to acquire audio data generated by the user.
[0115] The first validity detection module 20 is used to perform a first human voice validity detection on the audio data to obtain a first human voice validity ratio of the audio data.
[0116] The second validity detection module 30 is used to determine that the audio data is not in violation of the regulations when the first human voice validity ratio is less than a preset first ratio threshold; when the first human voice validity ratio is not less than the preset first ratio threshold, perform a second human voice validity detection on the audio data to obtain a second human voice validity score of the audio data.
[0117] The first violation determination module 40 is configured to determine that the audio data does not violate the rules when the second voice validity score is less than a preset second validity threshold.
[0118] The judgment module 50 is used to judge whether the audio data contains illegal sounds when the second human voice validity score is not less than a preset second validity threshold.
[0119] The second violation determination module 60 is configured to determine that the audio data violates a violation when the audio data contains a violation sound.
[0120] It can be seen from the above technical scheme that the present application obtains the audio data generated by the user and performs a first human voice validity detection on the audio data to obtain the first human voice validity ratio of the audio data; if the first human voice validity ratio is less than the preset first ratio threshold, it is determined that the audio data is not in violation of the rules; if the first human voice validity ratio is not less than the preset first ratio threshold, the second human voice validity detection is performed on the audio data to obtain the second human voice validity score of the audio data; if the second human voice validity score is less than the preset second validity threshold, it is determined that the audio data is not in violation of the rules; if the second human voice validity score is not less than the preset second validity threshold, it is determined whether the audio data contains illegal sounds; if the audio contains illegal sounds, it is determined that the audio data is in violation of the rules. This scheme performs a first human voice validity detection on the audio data generated by the user, and can obtain the first human voice validity ratio of the audio data, and compare the ratio with the preset first ratio threshold, so that it can directly determine whether the audio data contains human voices. If it does not contain human voices, it is not necessary to enter the subsequent judgment, which can shorten the detection process of the audio data. If it does, the second human voice validity test will be entered to make an accurate judgment on whether it contains human voice. If after two rounds of detection, the audio data is determined to contain no human voice, then it is determined that the audio data does not contain human voice, that is, it does not violate the rules. If the second round determines that it contains human voice, the audio data will be detected for illegal sounds. If the audio data contains illegal sounds, then the audio data is determined to be illegal, thereby improving the accuracy of the detection.
[0121] In one example, the first validity detection module 20 may include:
[0122] The conversion module is used to convert the audio data into a spectrogram.
[0123] The first human voice effectiveness score obtaining module is used to process the spectrogram using a pre-trained first human voice effectiveness detection model to obtain the first human voice effectiveness score of each frame of the audio data. The first human voice effectiveness detection model is trained using the spectrogram of the audio data sample as a training sample and the first human voice effectiveness score of each frame of the audio data sample as a sample label.
[0124] The first human voice effectiveness ratio determination module is used to obtain the first human voice effectiveness ratio of the audio data based on the first human voice effectiveness score of each frame of the audio data.
[0125] In one example, the first human voice effectiveness ratio determination module may include:
[0126] A comparison module is used to determine, for each frame of the audio data, that the frame of audio data does not contain a human voice if the first human voice validity score of the frame of audio data is less than a preset first validity threshold; and to determine that the frame of audio data contains a human voice if the first human voice validity score of the frame of audio data is not less than the first validity threshold.
[0127] The frame number determination module is used to obtain the total number of frames of the audio data and count the number of frames of the audio data containing human voice.
[0128] The frame number calculation module is used to determine the first human voice effectiveness ratio of the audio data based on the total number of frames of the audio data and the number of frames of the audio data containing the human voice.
[0129] In one example, the second validity detection module 30 may include:
[0130] The conversion module is used to convert the audio data into a spectrogram.
[0131] The second human voice effectiveness score obtaining module is used to process the spectrogram using a pre-trained second human voice effectiveness detection model to obtain the second human voice effectiveness score of the audio data; the second human voice effectiveness detection model is trained using the spectrogram of the audio data sample as a training sample and the second human voice effectiveness score of the audio data sample as a sample label.
[0132] In one example, the second voice effectiveness score obtaining module may include:
[0133] The first submodule is used to use the convolution module of the second human voice validity detection model to extract spatial features from the spectrum graph to obtain spatial feature information of the audio data.
[0134] The second submodule is used to use the loop module of the second human voice validity detection model to extract time domain features from the spatial feature information to obtain time domain feature information of the audio data.
[0135] The third sub-module is used to use the fully connected module of the second vocal effectiveness detection model to perform dimensionality adjustment on the time domain feature information to obtain the second vocal effectiveness sub-score of each frame of the audio data.
[0136] The fourth sub-module is used to use the pooling module of the second vocal effectiveness detection model to perform pooling processing on the second vocal effectiveness sub-score of each frame to obtain the second vocal effectiveness score of the audio data.
[0137] In one example, the determination module 50 may include:
[0138] The first violation score determination module is used to process the audio data using a preset violation detection model to obtain a first violation score corresponding to the audio data; the violation detection model is trained using audio data samples as training samples and the first violation score of the audio data as a sample label.
[0139] The illegal sound judgment module is used to judge that the audio data contains illegal sounds if the first illegal score of the audio data is not less than a preset first illegal threshold.
[0140] In one example, the apparatus may include:
[0141] The text data determination module is used to determine the text data corresponding to the audio data based on the audio data if the first violation score of the audio data is less than the first violation threshold, and judge whether the text data contains violation text.
[0142] The first illegal text judgment module is used to determine that the audio data does not violate the rules if the text data does not contain illegal text.
[0143] The second illegal text judgment module is used to determine that the audio data is illegal if the text data contains illegal text.
[0144] In one example, the text data determination module may include:
[0145] The matching module is used to perform keyword matching on the text data based on a preset illegal keyword word library. If the keyword matching is successful, it is determined that the text data contains illegal text.
[0146] The semantic analysis module is used to perform semantic analysis on the text data to obtain a second violation score of the audio data if the keyword matching is unsuccessful.
[0147] The module for determining that the text data does not contain any illegal text is configured to determine that the text data does not contain any illegal text if the second illegal score is less than a preset second illegal threshold.
[0148] A violation text determination module is included, which is used to determine that the text data contains violation text if the second violation score is not less than the second violation score.
[0149] Furthermore, an embodiment of the present application provides an audio detection device. Optionally, Figure 4 The hardware structure diagram of the audio detection device is shown in FIG. Figure 4 The hardware structure of the audio detection device may include: at least one processor 01 , at least one communication interface 02 , at least one memory 03 and at least one communication bus 04 .
[0150] In the embodiment of the present application, the number of the processor 01 , the communication interface 02 , the memory 03 , and the communication bus 04 is at least one, and the processor 01 , the communication interface 02 , and the memory 03 communicate with each other through the communication bus 04 .
[0151] The processor 01 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.
[0152] The memory 03 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory.
[0153] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to execute the following audio detection method, including:
[0154] Get audio data generated by the user;
[0155] Performing a first human voice validity detection on the audio data to obtain a first human voice validity ratio of the audio data;
[0156] If the first human voice validity ratio is less than a preset first ratio threshold, it is determined that the audio data is not in violation; if the first human voice validity ratio is not less than the preset first ratio threshold, a second human voice validity detection is performed on the audio data to obtain a second human voice validity score of the audio data;
[0157] If the second human voice validity score is less than a preset second validity threshold, determining that the audio data does not violate the rules;
[0158] If the second human voice validity score is not less than a preset second validity threshold, determining whether the audio data contains an illegal sound;
[0159] If the audio contains illegal sounds, the audio data is determined to be illegal. Optionally, the detailed functions and extended functions of the program can refer to the description of the audio detection method in the method embodiment.
[0160] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, and when the program is executed, controls the device where the storage medium is located to execute the following audio detection method, including:
[0161] Get audio data generated by the user;
[0162] Performing a first human voice validity detection on the audio data to obtain a first human voice validity ratio of the audio data;
[0163] If the first human voice validity ratio is less than a preset first ratio threshold, it is determined that the audio data is not in violation; if the first human voice validity ratio is not less than the preset first ratio threshold, a second human voice validity detection is performed on the audio data to obtain a second human voice validity score of the audio data;
[0164] If the second human voice validity score is less than a preset second validity threshold, determining that the audio data does not violate the rules;
[0165] If the second human voice validity score is not less than a preset second validity threshold, determining whether the audio data contains an illegal sound;
[0166] If the audio contains illegal sounds, the audio data is determined to be illegal. Specifically, the storage medium can be a computer-readable storage medium, and the computer-readable storage medium can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM.
[0167] Optionally, the detailed functions and extended functions of the program may refer to the description of the audio detection method in the method embodiment.
[0168] In addition, the functional modules in the various embodiments of the present disclosure can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a live broadcast device, or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present disclosure.
[0169] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0170] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0171] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An audio detection method, characterized in that: include: Get audio data generated by the user; Performing a first human voice validity detection on the audio data to obtain a first human voice validity ratio of the audio data; including: converting the audio data into a spectrogram; processing the spectrogram using a pre-trained first human voice validity detection model to obtain a first human voice validity score for each frame of the audio data; the first human voice validity detection model is trained using the spectrogram of the audio data sample as a training sample and the first human voice validity score of each frame of the audio data sample as a sample label; obtaining the first human voice validity ratio of the audio data based on the first human voice validity score of each frame of the audio data; If the first human voice effectiveness ratio is less than a preset first ratio threshold, it is determined that the audio data is not in violation of the rules; if the first human voice effectiveness ratio is not less than the preset first ratio threshold, a second human voice effectiveness detection is performed on the audio data to obtain a second human voice effectiveness score of the audio data; including: converting the audio data into a spectrogram; processing the spectrogram using a pre-trained second human voice effectiveness detection model to obtain a second human voice effectiveness score of the audio data; the second human voice effectiveness detection model is trained using the spectrogram of the audio data sample as a training sample and the second human voice effectiveness score of the audio data sample as a sample label; If the second human voice validity score is less than a preset second validity threshold, determining that the audio data does not violate the rules; If the second human voice validity score is not less than a preset second validity threshold, determining whether the audio data contains an illegal sound; If the audio data contains illegal sounds, it is determined that the audio data is illegal.
2. The method according to claim 1, characterized in that The obtaining of the first human voice effectiveness ratio of the audio data based on the first human voice effectiveness score of each frame of the audio data includes: For each frame of the audio data, if the first human voice validity score of the frame of audio data is less than a preset first validity threshold, it is determined that the frame of audio data does not contain a human voice; if the first human voice validity score of the frame of audio data is not less than the first validity threshold, it is determined that the frame of audio data contains a human voice; Obtaining the total number of frames of the audio data, and counting the number of frames of the audio data containing human voice; Based on the total number of frames of the audio data and the number of frames of the audio data containing human voice, a first human voice effectiveness ratio of the audio data is determined.
3. The method according to claim 1, characterized in that: The using a pre-trained second human voice validity detection model to process the spectrogram to obtain a second human voice validity score of the audio data includes: Using the convolution module of the second human voice validity detection model, extracting spatial features from the spectrogram to obtain spatial feature information of the audio data; Using the loop module of the second human voice validity detection model, extracting time domain features from the spatial feature information to obtain time domain feature information of the audio data; Using the fully connected module of the second human voice effectiveness detection model, dimensionally adjust the time domain feature information to obtain a second human voice effectiveness sub-score for each frame of the audio data; The second vocal effectiveness sub-score of each frame is pooled using the pooling module of the second vocal effectiveness detection model to obtain the second vocal effectiveness score of the audio data.
4. The method according to claim 1, characterized in that: The determining whether the audio data contains illegal sounds includes: Processing the audio data using a preset violation detection model to obtain a first violation score corresponding to the audio data; the violation detection model is trained using the audio data sample as a training sample and the first violation score of the audio data as a sample label; If the first violation score of the audio data is not less than a preset first violation threshold, it is determined that the audio data contains an illegal sound.
5. The method according to claim 4, characterized in that Also includes: If the first violation score of the audio data is less than the first violation threshold, determining text data corresponding to the audio data based on the audio data, and judging whether the text data contains a violation text; If the text data does not contain any illegal text, then determining that the audio data does not violate the rules; If the text data contains illegal text, the audio data is determined to be illegal.
6. The method according to claim 5, characterized in that The determining whether the text data contains illegal text includes: Perform keyword matching on the text data based on a preset illegal keyword word library, and if the keyword matching is successful, determine that the text data contains illegal text; If the keyword matching is unsuccessful, semantic analysis is performed on the text data to obtain a second violation score of the audio data; If the second violation score is less than a preset second violation threshold, determining that the text data does not contain a violation text; If the second violation score is not less than the second violation score, it is determined that the text data contains violation text.
7. An audio detection device, characterized in that: include: An acquisition module, used to acquire audio data generated by a user; A first validity detection module, configured to perform a first human voice validity detection on the audio data to obtain a first human voice validity ratio of the audio data; A second validity detection module is used to determine that the audio data is not in violation of the rules when the first human voice validity ratio is less than a preset first ratio threshold; and to perform a second human voice validity detection on the audio data to obtain a second human voice validity score of the audio data when the first human voice validity ratio is not less than the preset first ratio threshold; A first violation determination module, configured to determine that the audio data does not violate the rule when the second voice validity score is less than a preset second validity threshold; A judgment module, configured to judge whether the audio data contains an illegal sound when the second human voice validity score is not less than a preset second validity threshold; A second violation determination module, configured to determine that the audio data violates a violation when the audio contains a violation sound; The first validity detection module includes: a conversion module for converting the audio data into a spectrogram; a first voice validity score obtaining module for processing the spectrogram using a pre-trained first voice validity detection model to obtain a first voice validity score for each frame of the audio data; the first voice validity detection model is trained using the spectrogram of the audio data sample as a training sample and the first voice validity score of each frame of the audio data sample as a sample label; a first voice validity ratio determination module is used to obtain the first voice validity ratio of the audio data based on the first voice validity score of each frame of the audio data; The second validity detection module includes: a conversion module, used to convert the audio data into a spectrogram; a second vocal validity score obtaining module, used to process the spectrogram using a pre-trained second vocal validity detection model to obtain a second vocal validity score of the audio data; the second vocal validity detection model is trained using the spectrogram of the audio data sample as a training sample and the second vocal validity score of the audio data sample as a sample label.
