Sound event detection model training method and device, equipment and storage medium

Abstract

The invention relates to a sound event detection model training method and device, equipment and a storage medium. The method comprises the following steps: inputting a sample audio into a pre-trained sound event positioning model to obtain a prediction label of the sample audio; the sample audio is labeled with an initial label, and the initial label is used for representing whether the sample audio contains a sound event; the prediction label comprises probability distribution information that each audio frame in the sample audio comprises a sound event; cutting out an audio clip conforming to the target audio length from the sample audio, and marking a clip tag corresponding to the audio clip according to the initial tag and the prediction tag; the fragment label is used for representing whether the audio fragment contains a sound event; and according to the audio clip and the clip tag corresponding to the audio clip, training a to-be-trained sound event detection model to obtain a pre-trained sound event detection model. By adopting the method, the training efficiency of the sound event detection model can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a training method, apparatus, computer equipment, computer-readable storage medium, and computer program product for a sound event detection model. Background Art

[0002] With the increasing popularity of the Internet of Things (IoT) and smart devices, the demand for identifying sound events in audio signals is growing. Sound events refer to information about everyday environments and the physical events that occur within them, such as human voices, various animal sounds, natural sounds, and mechanical sounds. In areas such as smart homes and security monitoring, accurately identifying sound events such as speech and alarms provides powerful technical support for voice control in smart homes and fault warnings for industrial equipment. Consequently, sound event detection models based on architectures such as convolutional neural networks and recurrent neural networks have emerged. These models, based on large amounts of labeled training data, can automatically learn audio features and identify various sound events.

[0003] However, traditional sound event detection models are sensitive to the length of the input audio. When the audio length of the actual application scenario does not match the audio length of the training data, the training data needs to be manually re-labeled to adapt to the scenario, resulting in low training efficiency of the sound event detection model. Summary of the Invention

[0004] Based on this, it is necessary to provide a sound event detection model training method, device, computer equipment, computer-readable storage medium and computer program product that can improve the training efficiency of the sound event detection model in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for training a sound event detection model, comprising:

[0006] Inputting a sample audio into a pre-trained sound event localization model to obtain a predicted label for the sample audio; the sample audio is annotated with an initial label, the initial label being used to indicate whether the sample audio contains a sound event; the predicted label includes probability distribution information of each audio frame in the sample audio containing the sound event;

[0007] Cutting an audio segment that meets the target audio length from the sample audio, and labeling the audio segment with a segment label corresponding to the audio segment based on the initial label and the predicted label; the segment label is used to indicate whether the audio segment contains the sound event;

[0008] The sound event detection model to be trained is trained according to the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model.

[0009] In one embodiment, labeling the audio segment with a segment label corresponding to the audio segment based on the initial label and the predicted label includes:

[0010] In a case where the initial label indicates that the sample audio does not contain the sound event, marking the segment label corresponding to the audio segment as a first label; the first label is used to indicate that the audio sample does not contain the sound event;

[0011] In the case where the initial label represents that the sample audio contains the sound event, the probability distribution information of each audio frame in the audio segment containing the sound event is extracted from the predicted label of the sample audio as the truncated label, and the segment label corresponding to the audio segment is marked according to the number of probabilities in the truncated label that is greater than the probability threshold.

[0012] In one embodiment, labeling the segment label corresponding to the audio segment according to the number of probabilities greater than a probability threshold among the probabilities included in the predicted label includes:

[0013] Obtaining the number of probabilities in the truncated labels that are greater than a probability threshold;

[0014] When the probability number is greater than the first number threshold, marking the segment label corresponding to the audio segment as a second label; the second label is used to represent the sound event;

[0015] discarding the audio segment if the probability number is less than or equal to the first number threshold and greater than or equal to a second number threshold; the second number threshold is less than the first number threshold;

[0016] When the probability number is less than a second number threshold, the segment label corresponding to the audio segment is marked as the first label.

[0017] In one embodiment, the pre-trained sound event localization model includes a first backbone network and a downstream sound event localization model; inputting the sample audio into the pre-trained sound event localization model to obtain a predicted label for the sample audio includes:

[0018] Inputting the sample audio into the first backbone network to obtain acoustic features of the sample audio; the first backbone network is trained based on unlabeled audio of the same length as the sample audio;

[0019] The acoustic features of the sample audio are input into the downstream sound event localization model to obtain a predicted label of the sample audio.

[0020] In one embodiment, the sound event detection model to be trained includes a second backbone network and a downstream sound event detection model; and the sound event detection model to be trained is trained based on the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model, including:

[0021] Inputting the audio segment into a second backbone network to obtain acoustic features of the audio segment; the second backbone network is trained based on unlabeled audio of the same length as the target audio;

[0022] The downstream sound event detection model is trained according to the acoustic features of the audio segment and the segment label corresponding to the acoustic features to obtain the pre-trained sound event detection model.

[0023] In a second aspect, the present application further provides a sound event detection method, comprising:

[0024] Inputting the audio to be detected into the pre-trained sound event detection model to obtain a sound event detection result; the sound event detection result is used to indicate whether the audio to be detected contains a sound event;

[0025] The pre-trained sound event detection model is trained using the training method of the sound event detection model described above.

[0026] In a third aspect, the present application further provides a training device for a sound event detection model, comprising:

[0027] An input module is configured to input a sample audio into a pre-trained sound event localization model to obtain a predicted label for the sample audio; the sample audio is annotated with an initial label, the initial label being used to indicate whether the sample audio contains a sound event; the predicted label includes probability distribution information of each audio frame in the sample audio containing the sound event;

[0028] a labeling module, configured to crop an audio segment of a target audio length from the sample audio, and label the audio segment with a segment label corresponding to the audio segment based on the initial label and the predicted label; the segment label is used to indicate whether the audio segment contains the sound event;

[0029] The training module is used to train the sound event detection model to be trained based on the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model.

[0030] In a fourth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0031] In a fifth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0032] In a sixth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0033] The training method, apparatus, computer equipment, computer-readable storage medium and computer program product of the above-mentioned sound event detection model input the sample audio into a pre-trained sound event localization model to obtain a predicted label for the sample audio, wherein the sample audio is annotated with an initial label, and the initial label is used to characterize whether the sample audio contains a sound event, wherein the predicted label includes probability distribution information of whether each audio frame in the sample audio contains a sound event; then an audio segment that meets the target audio length is cropped from the sample audio, and a segment label corresponding to the audio segment is annotated according to the initial label and the predicted label, and the segment label is used to characterize whether the audio segment contains a sound event; based on the audio segment and the segment label corresponding to the audio segment, the sound event detection model to be trained is trained to obtain a pre-trained sound event detection model; in this way, through the pre-trained sound event localization model, the sound event detection model can be trained based on the labeled The method automatically generates a predicted label for the sample audio of the sample audio; then, an audio segment that is adapted to the target audio length of the actual application scenario is cut out from the existing sample audio, and the audio segment that meets the target audio length is automatically labeled according to the overall audio level information contained in the initial label and the frame-level probability distribution information contained in the predicted label. Then, the sound event detection model to be trained is trained based on the audio segment and the segment label corresponding to the audio segment, so that when the audio length of the actual application scenario does not match the audio length of the training data, the audio segment that meets the target audio length can be automatically labeled, thereby improving the training efficiency of the sound event detection model. In addition, the training method can adapt to different target audio length requirements, thereby improving the versatility and portability of the sound event detection model in multiple scenarios, thereby being able to support diverse data scenarios and detection requirements in sound event detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A diagram illustrating an application environment for a training method for a sound event detection model according to an embodiment;

[0036] Figure 2 1 is a flow chart of a method for training a sound event detection model in one embodiment;

[0037] Figure 3 A flowchart of a method for training a sound event detection model in one embodiment;

[0038] Figure 4 is a flowchart of a method for training a sound event detection model in another embodiment;

[0039] Figure 5 1 is a flow chart of a sound event detection method according to an embodiment;

[0040] Figure 6 is a structural block diagram of a training device for a sound event detection model in one embodiment;

[0041] Figure 7 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] In related technologies, the sound event detection model is sensitive to the length of the input audio. When the audio length of the actual application scenario does not match the audio length of the training data, it is necessary to manually re-label the training data to adapt to the scenario in order to retrain the model.

[0044] Traditional sound event detection methods directly pad the audio with zeros or repeat padding operations to unify the audio length of the training data to the audio length of the actual application scenario. However, this method may introduce invalid padding data, causing feature distortion and increased model calculation, thereby reducing the accuracy and efficiency of sound event detection.

[0045] In addition, if the audio length of the actual application scenario is short, such as less than 2 seconds, there will be many difficulties if the short audio needs to be manually re-annotated: it is difficult for humans to accurately determine whether the short audio contains the target sound event; humans cannot accurately locate the start and end time points of the event; the annotations of different annotators vary greatly, resulting in poor annotation consistency.

[0046] Therefore, in order to overcome the contradiction between the flexibility of model input length and the annotation cost, as well as the contradiction between the reliability of short-term audio manual annotation and the model training requirements, and in order to be able to train a model with high robustness and strong generalization ability under the conditions of limited labeled data, the present invention provides a training scheme for a sound event detection model.

[0047] The training method of the sound event detection model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The terminal 102 inputs the sample audio into a pre-trained sound event localization model to obtain a predicted label for the sample audio; the sample audio is annotated with an initial label, which is used to indicate whether the sample audio contains a sound event; the predicted label includes the probability distribution information of each audio frame in the sample audio containing a sound event; the terminal 102 crops an audio segment that meets the target audio length from the sample audio, and annotates the audio segment with a segment label corresponding to the audio segment based on the initial label and the predicted label; the segment label is used to indicate whether the audio segment contains a sound event; the terminal 102 trains the sound event detection model to be trained based on the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0048] In an exemplary embodiment, Figure 2 As shown, a training method for a sound event detection model is provided, and the method is applied to Figure 1 The terminal 102 in FIG. 1 is used as an example for explanation, including:

[0049] Step S202: Input the sample audio into a pre-trained sound event localization model to obtain a predicted label of the sample audio.

[0050] The sample audio can refer to manually annotated training data, where the audio length of the sample audio is inconsistent with the audio length of the actual application scenario. For example, the sample audio is 10 seconds long, but the actual application scenario requires detecting whether there is a sound event in 3 seconds of audio. Therefore, the dataset needs to be re-annotated to adapt to the 3-second audio length.

[0051] In actual applications, depending on the different sound event detection requirements, the sample audio can be audio collected in different application scenarios, such as home appliance control scenarios and security alarm scenarios in the smart home field, equipment fault monitoring scenarios in the industrial Internet of Things field, sleep monitoring scenarios in the medical and health field, and real-time interaction scenarios and content review scenarios in the voice social field. For example, in the voice social field that realizes real-time voice interaction, by detecting sound events, it is possible to detect illegal sounds in the voice to review the voice content, or detect applause and laughter during the voice interaction process to trigger interactive functions such as virtual gifts and emoticon effects. The large amount of sample audio collected in specific application scenarios needs to maintain data diversity to improve the generalization ability of the model. For example, the sample audio contains different environmental noises, different sound event durations, and different sound source distances.

[0052] The audio sample is annotated with an initial label, which is used to indicate whether the audio sample contains a sound event. In practical applications, each audio sample can be annotated with an initial label. This initial label can be a manually annotated global label. This initial label is an overall label that indicates whether the entire audio sample contains a certain sound event, that is, an audio-level label.

[0053] For example, for the case where there is only one type of sound event to be detected (such as the sound event is an illegal sound), assuming that each sample audio is 10 seconds, the initial label may include 0 or 1, 0 is used to indicate that the entire 10-second sample audio does not contain illegal sounds, and 1 is used to indicate that the entire 10-second sample audio contains illegal sounds. For the case where there are multiple types of sound events to be detected (such as dog barking, pig meowing, and cat meowing), assuming that each sample audio is 10 seconds, the initial label may include dog, pig, cat, and null. Dog is used to indicate that the entire 10-second sample audio contains dog barking, pig is used to indicate that the entire 10-second sample audio contains pig meowing, cat is used to indicate that the entire 10-second sample audio contains cat meowing, and null is used to indicate that the entire 10-second sample audio does not contain any of the above sound events.

[0054] Among them, the Sound Event Localization (SEL) model is a deep learning model based on weak supervision, which can be trained through coarse-grained supervision signals (such as audio-level labels). The pre-trained sound event localization model can be used to infer fine-grained prediction results (such as frame-level labels).

[0055] In one embodiment, the above-mentioned sample audio and the initial labels corresponding to the above-mentioned sample audio can be used to train the sound event localization model to be trained, thereby obtaining a pre-trained sound event localization model. The method for training the sound event localization model to be trained is a weakly supervised training method, which does not require knowing the specific start and end time of the sound event, and only requires audio-level labels (i.e., initial labels). Optionally, the weakly supervised training method can adopt a multi-instance learning (MIL) mechanism to associate frame-level predictions with global labels through maximum pooling or an attention mechanism, thereby training a pre-trained sound event localization model.

[0056] In the specific implementation, the pre-trained sound event localization model is adapted to the audio length of the sample audio. In other words, for any audio that meets the audio length of the sample audio, the pre-trained sound event localization model can output the probability distribution information of each audio frame in these audios containing sound events as a predicted label.

[0057] The predicted labels include the probability distribution information of each audio frame in the sample audio containing a sound event. The duration of each audio frame is not limited, for example, it can be equal to 0.1 or 0.2 seconds.

[0058] The probability distribution information can refer to a sequence of probabilities across all audio frames, reflecting the distribution of sound events along the timeline. This probability distribution information can be used to indirectly reflect the starting and ending points of a sound event. For example, if the probabilities of frames 5 to 10 are all greater than a certain threshold, the specific time period of the sound event can be considered to be between frames 5 and 10.

[0059] In specific implementations, a pre-trained sound event localization model can be used to make frame-by-frame predictions for sample audio that has been annotated with initial labels, outputting the probability distribution of each frame containing a sound event. In other words, the sound event localization model predicts a probability (0–1) for each audio frame, representing the likelihood that the audio frame contains a sound event. This probability can be considered the frame-level label for each audio frame. The sequence of these probabilities serves as the probability distribution information, i.e., the predicted label for the sample audio. This predicted label can be a pseudo-label, meaning a label that is not manually annotated.

[0060] For example, assuming that each audio frame is 0.1 seconds, a 10-second sample audio is divided into 100 audio frames. Since each audio frame corresponds to a probability value, the probability distribution information of the sample audio can be a sequence of 100 probability values.

[0061] Step S204 : cropping an audio segment that meets the target audio length from the sample audio, and labeling the audio segment with a segment label according to the initial label and the predicted label.

[0062] The target audio length is the audio length that needs to be detected in actual application scenarios. In specific implementations, the sample audio can be randomly cropped to obtain audio segments that meet the target audio length. For example, if the sample audio is 10 seconds and the target audio length is 3 seconds, a segment of 3.1 to 6.1 seconds, 5.6 to 8.6 seconds, or 7 to 10 seconds can be sampled from the 10-second sample audio to serve as the audio segment that meets the target audio length.

[0063] As an example, based on the initial label and the predicted label, the segment label corresponding to the audio segment can be marked by first determining whether the initial label indicates that the entire sample audio segment contains a sound event. If the initial label indicates that the entire sample audio segment contains a sound event, then based on the probability distribution information in the predicted label, determine whether the probability of N consecutive audio frames in the audio segment exceeds a first threshold (for example, 0.7); if not, the segment label corresponding to the audio segment is marked as the first label, and the first label is used to indicate that the audio segment does not contain a sound event; if so, the segment label corresponding to the audio segment is marked as the second label, and the second label is used to indicate that the audio segment contains a sound event. If the initial label indicates that the entire sample audio segment does not contain a sound event, then the segment label corresponding to the audio segment is directly marked as the first label. The number N can be determined based on the minimum duration of the sound event actually detected.

[0064] As another example, when labeling an audio segment with a segment label according to an initial label and a predicted label, it is possible to first determine whether the initial label indicates that the entire sample audio segment contains a sound event. If the initial label indicates that the entire sample audio segment contains a sound event, then based on the probability distribution information in the predicted label, determine whether the probability average of all audio frames in the audio segment exceeds a second threshold (for example, 0.5); if not, then the segment label corresponding to the audio segment is labeled as the first label, and the first label is used to indicate that the audio segment does not contain a sound event; if it exceeds, then the segment label corresponding to the audio segment is labeled as the second label, and the second label is used to indicate that the audio segment contains a sound event. If the initial label indicates that the entire sample audio segment does not contain a sound event, then the segment label corresponding to the audio segment is directly labeled as the first label.

[0065] As another example, when labeling an audio segment with a segment label corresponding to the initial label and the predicted label, it is possible to first determine whether the initial label indicates that the entire sample audio segment contains a sound event. If the initial label indicates that the entire sample audio segment contains a sound event, then based on the probability distribution information in the predicted label, determine whether the ratio of the number of audio frames in the audio segment whose probability exceeds the third threshold to the total number of audio frames exceeds the third threshold (for example, 0.6); if not, then the segment label corresponding to the audio segment is labeled as the first label, and the first label is used to indicate that the audio segment does not contain a sound event; if it exceeds, then the segment label corresponding to the audio segment is labeled as the second label, and the second label is used to indicate that the audio segment contains a sound event. If the initial label indicates that the entire sample audio segment does not contain a sound event, then the segment label corresponding to the audio segment is directly labeled as the first label.

[0066] The segment label is used to indicate whether an audio segment contains a sound event. This segment label is a pseudo-label automatically generated based on the initial and predicted labels, enabling automatic annotation of audio segments that meet the target audio length. If the target audio length of the audio segment is shorter, such as 2 seconds, automatic annotation of short audio segments can be achieved, avoiding the low accuracy associated with manual annotation of short audio segments.

[0067] Therefore, by automatically generating segment labels and training the sound event detection model based on audio segments and the segment labels corresponding to the audio segments, it can adapt to input audio of different lengths, improve the versatility and portability of the model in multiple scenarios and multiple projects, and at the same time, greatly reduce the dependence on fine-grained annotations (event start and end times), save annotation costs and cycles, and through the automated pseudo-annotation generation process, the utilization value of "weakly labeled" data can be improved, thereby supporting diverse data scenarios and actual needs in sound event detection tasks.

[0068] For example, for the case where there is only one type of sound event to be detected (such as the sound event is an illegal sound), assuming that each audio clip is 3 seconds, the clip label can include 0 or 1, 0 is used to indicate that the entire 3-second audio clip does not contain illegal sounds, and 1 is used to indicate that the entire 3-second audio clip contains illegal sounds. For the case where there are multiple types of sound events to be detected (such as dog barking, pig meowing, and cat meowing), assuming that each audio clip is 3 seconds, the clip label can include dog, pig, cat, and null. Dog is used to indicate that the entire 3-second audio clip contains dog barking, pig is used to indicate that the entire 3-second audio clip contains pig meowing, cat is used to indicate that the entire 3-second audio clip contains cat meowing, and null is used to indicate that the entire 3-second audio clip does not contain any of the above sound events.

[0069] Step S206: Train the sound event detection model to be trained based on the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model.

[0070] In the specific implementation, after determining the segment label, the sound event detection model to be trained can be trained through supervised learning training method based on the audio segment and the segment label corresponding to the audio segment. First, the audio segment is input into the sound event detection model to be trained to obtain the model prediction label; then the training loss (such as cross entropy loss, etc.) is calculated based on the difference between the model prediction label and the segment label; and the sound event detection model to be trained is trained based on the training loss to obtain a pre-trained sound event detection model.

[0071] Among them, the Sound Event Detection (SED) model is a deep learning model adapted to the target audio length. It can be used to detect whether any audio that meets the target audio length contains sound events, so that audio clips that meet the target audio length can be automatically labeled, and the sound event detection model to be trained is trained based on the audio clips and their corresponding clip labels, realizing adaptive training for the target audio length without manual re-labeling, so that the pre-trained sound event detection model can meet the needs of sound event detection for any audio that meets the target audio length, thereby improving the training efficiency of the sound event detection model. Moreover, compared with the training method in the prior art that directly performs zero-padding or repeated padding operations on audio, the training method of the sound event detection model generates frame-level probability distribution information for the original sample audio, retains the time-frequency structure of the real audio signal, and then cuts out actual and valid audio clips from the sample audio, and then labels the audio clips based on the initial label and the predicted label, avoiding the destruction of acoustic features caused by artificial zero-padding or padding, ensuring the authenticity of the acoustic features of the training data, and improving the training accuracy of the sound event detection model, thereby improving the detection accuracy of the pre-trained sound event detection model for sound events.

[0072] In actual applications, when there is only one sound event category, the sound event detection model can output whether the audio that meets the target audio length contains the sound event; when there are multiple sound event categories, the sound event detection model can output whether the audio that meets the target audio length contains a specific category of sound event.

[0073] In the training method of the above-mentioned sound event detection model, the sample audio is input into the pre-trained sound event localization model to obtain a predicted label of the sample audio, and the sample audio is annotated with an initial label, and the initial label is used to characterize whether the sample audio contains a sound event, wherein the predicted label includes the probability distribution information of each audio frame in the sample audio containing a sound event; then an audio segment that meets the target audio length is cropped from the sample audio, and a segment label corresponding to the audio segment is annotated according to the initial label and the predicted label, and the segment label is used to characterize whether the audio segment contains a sound event; based on the audio segment and the segment label corresponding to the audio segment, the sound event detection model to be trained is trained to obtain a pre-trained sound event detection model; in this way, the pre-trained sound event localization model can be used to automatically generate a sample audio based on the labeled sample audio. The predicted label of the frequency; then, an audio clip that is adapted to the target audio length of the actual application scenario is cut out from the existing sample audio, and the audio clip that meets the target audio length is automatically labeled according to the overall audio level information contained in the initial label and the frame-level probability distribution information contained in the predicted label. Then, the sound event detection model to be trained is trained based on the audio clips and the clip labels corresponding to the audio clips. Therefore, when the audio length of the actual application scenario does not match the audio length of the training data, the audio clip that meets the target audio length can be automatically labeled, thereby improving the training efficiency of the sound event detection model. In addition, the training method can adapt to different target audio length requirements, thereby improving the versatility and portability of the sound event detection model in multiple scenarios, thereby being able to support diverse data scenarios and detection requirements in sound event detection tasks.

[0074] In another embodiment, the audio segment is labeled with a segment label corresponding to the audio segment based on the initial label and the predicted label, including: when the initial label represents that the sample audio does not contain a sound event, the segment label corresponding to the audio segment is labeled as a first label; when the initial label represents that the sample audio contains a sound event, the probability distribution information of each audio frame in the audio segment containing the sound event is extracted from the predicted label of the sample audio as a truncated label, and the segment label corresponding to the audio segment is labeled according to the number of probabilities in the truncated label that is greater than a probability threshold.

[0075] The first tag is used to represent an event that does not contain sound.

[0076] In the specific implementation, when the initial label represents that the sample audio does not contain a sound event, the segment label corresponding to the marked audio segment is marked as the first label, which means that when the initial label represents that the sample audio does not contain a sound event, it is directly determined that the audio segments extracted from the sample audio do not contain a sound event, and the audio segment is automatically marked as the first label.

[0077] In a specific implementation, when the initial label indicates that the sample audio contains a sound event, the coarse-grained label of the long audio indicates that the sample audio contains a sound event. Then, whether the cropped audio segment contains a sound event can be further judged to generate a fine-grained label for the short audio. When the initial label indicates that the sample audio contains a sound event, the probability distribution information of each audio frame in the audio segment containing the sound event is extracted from the predicted label of the sample audio as the truncated label. For example, each audio frame is 0.1 seconds, and a 10-second sample audio is divided into 100 audio frames. Since each audio frame corresponds to a probability value, the probability distribution information of the sample audio can be a sequence of 100 probability values. If a 3-second audio segment between 3 seconds and 6 seconds is cropped, the probability value between 3 seconds and 6 seconds can be extracted from the predicted label as the probability distribution information of each audio frame in the audio segment containing a sound event. The probability distribution information of the audio segment includes a sequence of 30 probability values; then the probability distribution information of the audio segment is used as the truncated label corresponding to the audio segment.

[0078] In a specific implementation, the system determines whether each probability included in the truncated label is greater than a probability threshold and counts the number of probabilities that are greater than the probability threshold (for example, the probability threshold could be 0.7). The audio segment is then labeled based on the probability count, or based on the ratio of the probability count to the total probability count (i.e., the total number of audio frames in the audio segment).

[0079] As an example, when the probability number is greater than the first quantity threshold, the segment label corresponding to the audio segment can be marked as the second label; the second label is used to characterize the sound event; when the probability number is less than the first quantity threshold, the segment label corresponding to the audio segment can be marked as the first label.

[0080] As another example, the ratio between the number of probabilities greater than a probability threshold and the total number of probabilities can be calculated, and when the ratio is greater than a first probability threshold, the segment label corresponding to the audio segment is marked as the second label; when the ratio is less than or equal to the first probability threshold, the segment label corresponding to the audio segment is marked as the first label.

[0081] The technical solution of the above embodiment, when the initial label represents that the sample audio does not contain a sound event, directly determines that the audio segments extracted from the sample audio do not contain a sound event. Since the initial label has a high confidence level, it can avoid misjudging background noise as a sound event, thereby improving training efficiency and accuracy; when the initial label represents that the sample audio contains a sound event, it can be further verified whether the audio segment extracted from the sample audio contains a sufficient number of high-probability audio frames, and then the segment label is determined. The accuracy of the labeled segment label is improved through the dual judgment mechanism, thereby improving the training accuracy of the sound event detection model, and further improving the accuracy of the sound event detection model in detecting sound events.

[0082] In another embodiment, labeling the segment label corresponding to the audio segment according to the probability number greater than the probability threshold in the truncated label includes: obtaining the probability number greater than the probability threshold in the truncated label; when the probability number is greater than the first quantity threshold, labeling the segment label corresponding to the audio segment as the second label; the second label is used to characterize the sound event; when the probability number is less than or equal to the first quantity threshold, and greater than or equal to the second quantity threshold, discarding the audio segment; the second quantity threshold is less than the first quantity threshold; when the probability number is less than the second quantity threshold, labeling the segment label corresponding to the audio segment as the first label.

[0083] Exemplarily, the first quantity threshold and the second quantity threshold may be determined based on the total number of audio frames corresponding to the audio clip. For example, if the total number of audio frames is 100, the first quantity threshold may be 70 and the second quantity threshold may be 20.

[0084] In a specific implementation, when the probability number is greater than the first quantity threshold, it is considered that the audio clip contains a high possibility of a sound event, and therefore the segment label corresponding to the audio clip can be directly marked as the second label; when the probability number is less than or equal to the first quantity threshold, and greater than or equal to the second quantity threshold, the audio clip is discarded, and fuzzy samples can be actively discarded to improve the quality of the training samples; when the probability number is less than the second quantity threshold, it is considered that the audio clip contains a very small possibility of a sound event, and therefore the segment label corresponding to the audio clip can be directly marked as the first label.

[0085] The technical solution of this embodiment realizes the refined screening and automatic labeling of audio clips through a dual quantity threshold mechanism, thereby improving the quality of training data used to train the sound event detection model, thereby improving the training accuracy and efficiency of the sound event detection model.

[0086] Optionally, labeling the segment label corresponding to the audio segment based on the number of probabilities greater than the probability threshold in the truncated label may also include: calculating the ratio between the number of probabilities greater than the probability threshold and the total probability number (i.e., the total number of audio frames in the audio segment), and if the ratio is greater than a first probability threshold, labeling the segment label corresponding to the audio segment as the second label; if the ratio is less than or equal to the first probability threshold, discarding the audio segment; if the ratio is less than the second probability threshold, labeling the segment label corresponding to the audio segment as the first label, and the second probability threshold is less than the first probability threshold.

[0087] For example, during the training of the sound event detection model, several audio samples can be randomly sampled (the audio length of the sample audio is not the target audio length). For each audio sample, an audio segment of the target audio length is randomly cropped, and the corresponding manual label (i.e., initial label) and the pseudo label (predicted label) predicted by the sound event localization model are indexed. The segment label annotation can be divided into the following four cases:

[0088] 1. If the original manual label representation does not contain a certain sound event, the audio segment of the target audio length is marked as not containing the sound event, that is, the segment label is used to represent that it does not contain the sound event;

[0089] 2. If the original manual label represents a sound event, and the number of probabilities in the pseudo-labels (the probability sequence of audio frames containing a sound event) corresponding to the audio segment of the target audio length that is greater than the probability threshold is greater than a certain value, then the audio segment of the target audio length is marked as containing the sound event. That is, the segment label is used to represent the sound event.

[0090] 3. If the original manual label represents a sound event, and the number of pseudo labels corresponding to the audio clip of the target audio length (the probability sequence of audio frames containing a sound event) with a probability greater than the probability threshold is less than a certain value, then the sample is discarded;

[0091] 4. If the original manual label represents a sound event, and the number of probabilities in the pseudo-labels (the probability sequence of audio frames containing a sound event) corresponding to the audio segment of the target audio length is greater than the probability threshold is 0, then the audio segment of the target audio length is marked as not containing the sound event. That is, the segment label is used to represent that it does not contain the sound event.

[0092] For example, assuming the original sample audio is 10 seconds long, the target audio is 3 seconds long, and there is only one sound event category. The sound event localization model can output the probabilities corresponding to 100 audio frames for the 10-second sample audio, that is, there are 10 probabilities for 1 second, and each probability represents the predicted probability corresponding to a 0.1-second audio frame. The segment labeling can be divided into the following four cases:

[0093] 1. If the manual label is 0 (indicating that the sound event is not included), then no matter which audio segment is cut, its segment label will be 0 (indicating that the sound event is not included);

[0094] 2. If the manual label is 1, and an audio segment of 3.1 to 6.1 seconds is sampled, and 20 of the 30 probabilities are greater than the probability threshold, the audio segment of the target audio length is marked as 1 (indicating that it contains the sound event), that is, the segment label is 1;

[0095] 3. If the manual label is 1, and an audio clip of 5.6 to 8.6 seconds is sampled, and 5 of the 30 probabilities are greater than the probability threshold, the audio clip is discarded;

[0096] 4. If the manual label is 1, and a 7-10 second audio clip is sampled, and 0 of the 30 scores is greater than the probability threshold, the audio clip of the target audio length is marked as 0 (indicating that it does not contain the sound event), that is, the clip label is 0.

[0097] In another embodiment, the pre-trained sound event localization model includes a first backbone network and a downstream sound event localization model; inputting the sample audio into the pre-trained sound event localization model to obtain a predicted label of the sample audio includes: inputting the sample audio into the first backbone network to obtain the acoustic features of the sample audio; inputting the acoustic features of the sample audio into the downstream sound event localization model to obtain a predicted label of the sample audio.

[0098] In a specific implementation, the first backbone network can be a backbone network based on the EAT (Efficient Audio Transformer) architecture, which is an improved neural network based on the Transformer architecture.

[0099] The first backbone network is trained based on unlabeled audio data that is consistent with the audio length of the sample audio. For example, if the audio length of the sample audio is 10 seconds, the first backbone network can be trained based on a large amount of unlabeled 10-second audio data.

[0100] In a specific implementation, the first backbone network can be trained using a self-supervised learning method. Self-supervised learning does not require manually labeled training data and can mine supervisory information that can be applied to itself from large-scale unsupervised data, thereby learning from another part of the input data. The first backbone network is trained on large-scale unlabeled audio data, so that the sound event localization model can extract common acoustic features through the first backbone network and effectively adapt to audio inputs of different lengths. Among them, acoustic features can include common acoustic features such as timbre, rhythm, and spectral patterns.

[0101] After the first backbone network is trained, the parameters of the first backbone network learned through self-supervision can be fixed (the first backbone network can be frozen), and the downstream sound event localization model can be trained on this basis, thereby decoupling feature learning and task adaptation and improving the subsequent training efficiency of the sound event detection model.

[0102] Among them, the downstream sound event localization model is a sound event localization (SoundEvent Localization, SEL) model constructed based on a weakly supervised learning framework. The description of the downstream sound event localization model can refer to the description of the sound event localization model in the above embodiment. The downstream sound event localization model can be obtained by training using the above sample audio and the initial labels corresponding to the above sample audio.

[0103] In the specific implementation, the acoustic features extracted from the sample audio by the first backbone network are input into the downstream sound event localization model. The downstream sound event localization model can perform frame-level sound event prediction on the sample audio based on the acoustic features, and output the probability distribution information of the sound events contained in each frame to obtain the predicted label of the sample audio.

[0104] The technical solution of this embodiment uses the first backbone network to learn acoustic features specific to the audio length from unlabeled audio that is consistent with the audio length of the sample audio, thereby improving the training accuracy of the sound event detection model.

[0105] In another embodiment, the sound event detection model to be trained includes a second backbone network and a downstream sound event detection model; the sound event detection model to be trained is trained according to the audio clip and the clip label corresponding to the audio clip to obtain a pre-trained sound event detection model, including: inputting the audio clip into the second backbone network to obtain the acoustic features of the audio clip; according to the acoustic features of the audio clip and the clip label corresponding to the acoustic features, the downstream sound event detection model is trained to obtain a pre-trained sound event detection model.

[0106] In a specific implementation, the second backbone network can be a backbone network based on the EAT (Efficient Audio Transformer) architecture, which is an improved neural network based on the Transformer architecture.

[0107] The second backbone network is trained on unlabeled audio of the same length as the target audio. For example, if the audio clip is 3 seconds long, the second backbone network can be trained on a large amount of unlabeled 3-second audio data. This allows the second backbone network to learn acoustic feature representations specific to the target audio length.

[0108] In a specific implementation, the second backbone network can be trained using a self-supervised learning method. Training the second backbone network on large-scale unlabeled audio data enables the sound event detection model to extract universal acoustic features through the second backbone network and effectively adapt to audio inputs of varying lengths.

[0109] After training the second backbone network, the parameters of the second backbone network learned through self-supervision can be fixed (the second backbone network can be frozen), and the downstream sound event detection model can be trained on this basis, thereby decoupling feature learning and task adaptation and improving the training efficiency of the sound event detection model.

[0110] Among them, the downstream sound event detection model is a sound event detection (SED) model constructed based on a supervised learning framework. The description of the downstream sound event detection model can refer to the description of the sound event detection model in the above embodiment.

[0111] In the specific implementation, based on the acoustic features of the audio clip and the clip labels corresponding to the acoustic features, the downstream sound event detection model can be trained through supervised learning training. First, the acoustic features are input into the downstream sound event detection model to obtain the model prediction label; then the training loss (such as cross entropy loss, etc.) is calculated based on the difference between the model prediction label and the clip label; and the downstream sound event detection model is trained based on the training loss to obtain a pre-trained sound event detection model.

[0112] The technical solution of this embodiment improves the training accuracy of the sound event detection model by learning acoustic features specific to the target audio length from unlabeled audio with the same length as the target audio through the second backbone network; and, by dividing the problem of audio length adaptation training into feature extraction layer adaptation (second backbone network) and event localization layer adaptation (downstream sound event detection model), the training efficiency of the sound event detection model can be improved, which is suitable for application scenarios that require frequent adjustment of detection time (such as illegal audio detection on voice platforms).

[0113] In summary, the above embodiments propose a sound event detection model training method that is adaptable to inputs of different lengths. By combining self-supervised learning, weakly supervised training technology, and pseudo-label generation, it reduces manual labeling costs and improves the efficiency and effectiveness of model training.

[0114] To facilitate understanding by those skilled in the art, Figure 3A flowchart of a method for training a sound event detection model is provided as an example. First, a manually annotated dataset of audio dimensions (only annotating whether the audio contains a specific sound event) is used to obtain a sound event localization model through weakly supervised training. The sound event localization model performs frame-level event prediction on the manually annotated dataset to generate a pseudo-annotated dataset of frame dimensions, which contains pseudo-annotations of the start and end points of the sound events (i.e., the probability distribution information of the sound events, based on which the specific distribution location of the sound events can be known). Then, the audio in the manually annotated dataset is randomly segmented online into audio segments that meet the target audio length, and then annotation calculations are performed to generate segment labels corresponding to the audio segments. First, the self-supervised backbone network is trained using audio of the target audio length, and then the downstream sound event detection model connected to the backbone network is trained based on the audio segments and their corresponding segment labels to obtain a pre-trained sound event detection model.

[0115] The above embodiment proposes a training method for a sound event detection model that adapts to inputs of varying lengths. By combining self-supervised learning, weakly supervised training techniques, and pseudo-label generation, it reduces manual labeling costs and improves the efficiency and effectiveness of model training. This sound event detection model training method can adapt to input audio of varying lengths, improving the model's versatility and portability across multiple scenarios and projects. It significantly reduces reliance on fine-grained labeling (event start and end times), saving labeling costs and cycles. It automates the pseudo-label generation process, increasing the value of "weakly labeled" data. It supports diverse data scenarios and practical needs in sound event detection tasks.

[0116] In another embodiment, Figure 4 As shown, a training method for a sound event detection model is provided, and the method is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, including the following steps:

[0117] Step S402: Input the sample audio into the first backbone network to obtain the acoustic features of the sample audio.

[0118] The first backbone network is trained based on unlabeled audio with the same audio length as the sample audio.

[0119] Among them, the pre-trained sound event localization model includes a first backbone network and a downstream sound event localization model.

[0120] Step S404: Input the acoustic features of the sample audio into a downstream sound event localization model to obtain a predicted label of the sample audio.

[0121] The sample audio is annotated with an initial label, which is used to characterize whether the sample audio contains a sound event; the predicted label includes the probability distribution information of each audio frame in the sample audio containing a sound event.

[0122] Step S406: cropping an audio segment that meets the target audio length from the sample audio.

[0123] Step S408 , determining whether the initial tag indicates that the sample audio contains a sound event; if so, executing step S410 ; if not, executing step S412 .

[0124] In step S410, when the initial label indicates that the sample audio contains a sound event, the probability distribution information of each audio frame in the audio segment containing the sound event is extracted from the predicted label of the sample audio as the extracted label, and the segment label corresponding to the audio segment is marked according to the number of probabilities in the extracted label that is greater than the probability threshold, and step S414 is executed.

[0125] In one embodiment, labeling the segment label corresponding to the audio segment according to the probability number greater than the probability threshold in the truncated label includes: obtaining the probability number greater than the probability threshold in the truncated label; when the probability number is greater than the first quantity threshold, labeling the segment label corresponding to the audio segment as the second label; the second label is used to characterize the sound event; when the probability number is less than or equal to the first quantity threshold, and greater than or equal to the second quantity threshold, discarding the audio segment; the second quantity threshold is less than the first quantity threshold; when the probability number is less than the second quantity threshold, labeling the segment label corresponding to the audio segment as the first label.

[0126] Step S412 : When the initial label indicates that the sample audio does not contain a sound event, the segment label corresponding to the audio segment is marked as a first label.

[0127] The first tag is used to represent an event that does not contain sound.

[0128] Step S414: input the audio segment into the second backbone network to obtain the acoustic features of the audio segment.

[0129] Among them, the second backbone network is trained based on unlabeled audio with the same length as the target audio.

[0130] Step S416: Train the downstream sound event detection model based on the acoustic features of the audio segment and the segment labels corresponding to the acoustic features to obtain a pre-trained sound event detection model.

[0131] The sound event detection model to be trained includes a second backbone network and a downstream sound event detection model.

[0132] It should be noted that the specific limitations of the above steps can be found in the specific limitations of the training method for a sound event detection model mentioned above.

[0133] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the representations of the arrows, these steps are not necessarily executed in sequence in the order represented by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0134] Based on the same inventive concept, Figure 5 As shown, the embodiment of the present application also provides a sound event detection method, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, including the following steps:

[0135] Step S502: input the audio to be detected into a pre-trained sound event detection model to obtain a sound event detection result.

[0136] The sound event detection result is used to indicate whether the audio to be detected contains a sound event.

[0137] The pre-trained sound event detection model is trained using the training method of the sound event detection model in the above embodiment.

[0138] In a specific implementation, the audio to be detected matches the target audio length. The audio to be detected is then fed into a pre-trained sound event detection model trained to the target audio length, yielding a sound event detection result. Furthermore, the audio to be detected can be collected in specific application scenarios (such as real-time interactive scenarios and content review scenarios in voice-based social networking), and the pre-trained sound event detection model is trained based on training data collected in the same application scenario.

[0139] Based on the same inventive concept, embodiments of the present application also provide a sound event detection model training device for implementing the aforementioned sound event detection model training method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the one or more sound event detection model training device embodiments provided below can be found in the limitations of the sound event detection model training method described above and will not be further elaborated here.

[0140] In an exemplary embodiment, Figure 6 As shown, a training device for a sound event detection model is provided, comprising:

[0141] The input module 610 is used to input the sample audio into a pre-trained sound event localization model to obtain a predicted label of the sample audio; the sample audio is annotated with an initial label, and the initial label is used to characterize whether the sample audio contains a sound event; the predicted label includes the probability distribution information of each audio frame in the sample audio containing the sound event.

[0142] The labeling module 620 is used to crop an audio segment that meets the target audio length from the sample audio, and label the segment label corresponding to the audio segment based on the initial label and the predicted label; the segment label is used to indicate whether the audio segment contains the sound event.

[0143] The training module 630 is used to train the sound event detection model to be trained based on the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model.

[0144] In one embodiment, the labeling module 620 is specifically used to label the segment label corresponding to the audio segment as a first label when the initial label represents that the sample audio does not contain the sound event; the first label is used to represent that the sound event is not contained; when the initial label represents that the sample audio contains the sound event, the probability distribution information of each audio frame in the audio segment containing the sound event is extracted from the predicted label of the sample audio as a truncated label, and the segment label corresponding to the audio segment is labeled according to the number of probabilities in the truncated label that is greater than the probability threshold.

[0145] In one embodiment, the labeling module 620 is specifically used to obtain the probability number greater than the probability threshold in the truncated label; when the probability number is greater than the first quantity threshold, label the segment label corresponding to the audio segment as the second label; the second label is used to characterize the sound event; when the probability number is less than or equal to the first quantity threshold, and greater than or equal to the second quantity threshold, discard the audio segment; the second quantity threshold is less than the first quantity threshold; when the probability number is less than the second quantity threshold, label the segment label corresponding to the audio segment as the first label.

[0146] In one embodiment, the pre-trained sound event localization model includes a first backbone network and a downstream sound event localization model; the input module 610 is specifically used to input the sample audio into the first backbone network to obtain the acoustic features of the sample audio; the first backbone network is trained based on unlabeled audio with the same audio length as the sample audio; the acoustic features of the sample audio are input into the downstream sound event localization model to obtain the predicted label of the sample audio.

[0147] In one embodiment, the sound event detection model to be trained includes a second backbone network and a downstream sound event detection model; the training module 630 is specifically used to input the audio segment into the second backbone network to obtain the acoustic features of the audio segment; the second backbone network is trained based on unlabeled audio with the same length as the target audio; the downstream sound event detection model is trained based on the acoustic features of the audio segment and the segment label corresponding to the acoustic features to obtain the pre-trained sound event detection model.

[0148] Each module in the above-mentioned sound event detection device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in the form of software in a memory in the computer device, so that the processor can call and execute the corresponding operations of each module.

[0149] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for training a sound event detection model. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0150] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0151] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0153] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0155] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0156] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0157] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for training a sound event detection model, characterized in that: The method comprises: Inputting a sample audio into a pre-trained sound event localization model to obtain a predicted label for the sample audio; the sample audio is annotated with an initial label, the initial label being used to indicate whether the sample audio contains a sound event; the predicted label includes probability distribution information of each audio frame in the sample audio containing the sound event; Cutting an audio segment that meets the target audio length from the sample audio, and labeling the audio segment with a segment label corresponding to the audio segment based on the initial label and the predicted label; the segment label is used to indicate whether the audio segment contains the sound event; The sound event detection model to be trained is trained according to the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model.

2. The method according to claim 1, characterized in that The step of labeling the audio segment with a corresponding segment label according to the initial label and the predicted label includes: In a case where the initial label indicates that the sample audio does not contain the sound event, marking the segment label corresponding to the audio segment as a first label; the first label is used to indicate that the audio sample does not contain the sound event; In the case where the initial label represents that the sample audio contains the sound event, the probability distribution information of each audio frame in the audio segment containing the sound event is extracted from the predicted label of the sample audio as the truncated label, and the segment label corresponding to the audio segment is marked according to the number of probabilities in the truncated label that is greater than the probability threshold.

3. The method according to claim 2, characterized in that The step of labeling the audio segment corresponding to the audio segment according to the number of probabilities in the extracted labels that is greater than a probability threshold includes: Obtaining the number of probabilities greater than a probability threshold in the truncated labels; When the probability number is greater than the first number threshold, marking the segment label corresponding to the audio segment as a second label; the second label is used to represent the sound event; discarding the audio segment if the probability number is less than or equal to the first number threshold and greater than or equal to a second number threshold; the second number threshold is less than the first number threshold; When the probability number is less than a second number threshold, the segment label corresponding to the audio segment is marked as the first label.

4. The method according to claim 1, wherein The pre-trained sound event localization model includes a first backbone network and a downstream sound event localization model; inputting the sample audio into the pre-trained sound event localization model to obtain a predicted label of the sample audio includes: Inputting the sample audio into the first backbone network to obtain acoustic features of the sample audio; the first backbone network is trained based on unlabeled audio of the same length as the sample audio; The acoustic features of the sample audio are input into the downstream sound event localization model to obtain a predicted label of the sample audio.

5. The method according to claim 1, wherein The sound event detection model to be trained includes a second backbone network and a downstream sound event detection model; and the sound event detection model to be trained is trained based on the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model, including: Inputting the audio segment into a second backbone network to obtain acoustic features of the audio segment; the second backbone network is trained based on unlabeled audio of the same length as the target audio; The downstream sound event detection model is trained according to the acoustic features of the audio segment and the segment label corresponding to the acoustic features to obtain the pre-trained sound event detection model.

6. A sound event detection method, characterized in that: The method comprises: Inputting the audio to be detected into the pre-trained sound event detection model to obtain a sound event detection result; the sound event detection result is used to indicate whether the audio to be detected contains a sound event; The pre-trained sound event detection model is trained using the sound event detection model training method according to any one of claims 1 to 5.

7. A training device for a sound event detection model, characterized in that: The device comprises: An input module is configured to input a sample audio into a pre-trained sound event localization model to obtain a predicted label for the sample audio; the sample audio is annotated with an initial label, the initial label being used to indicate whether the sample audio contains a sound event; the predicted label includes probability distribution information of each audio frame in the sample audio containing the sound event; a labeling module, configured to crop an audio segment of a target audio length from the sample audio, and label the audio segment with a segment label corresponding to the audio segment based on the initial label and the predicted label; the segment label is used to indicate whether the audio segment contains the sound event; The training module is used to train the sound event detection model to be trained based on the audio segment and the segment label corresponding to the audio segment to obtain a pre-trained sound event detection model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images