Model generation methods, sound event detection methods, devices, media and equipment

Abstract

This disclosure relates to a model generation method, a sound event detection method, an apparatus, a medium, and a device. The method includes: extracting audio features from a first sample audio file and inputting the audio features into a sound event detection model to obtain a first detection result; determining the target loss of the model based on the first annotation result of the first sample audio file and the first detection result, the target loss including cross-entropy loss and continuity error penalty loss; and updating the model parameters based on the target loss. Since continuity detection errors are the main cause of misidentification, the continuity error penalty term is included as part of the loss function during the sound event detection model training process. The model trained using this method can maintain high accuracy while keeping the sound event recall rate unchanged. That is, it can significantly reduce misidentifications that affect user experience without introducing more missed detections, thereby improving the accuracy of sound event detection and user experience.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, specifically to a model generation method, a sound event detection method, an apparatus, a medium, and a device. Background Technology

[0002] The goal of sound event detection is to detect all sound events occurring in an audio segment. This requires identifying specific sound event categories and providing the start and end times of each type of sound event. Sound event detection is widely used in various fields, such as wildlife detection and intelligent audio captioning. Most fields require extremely high accuracy for sound event detection models. For example, in intelligent captioning, insertion errors (i.e., misidentification, mistaking other sounds for the target sound event) have a far greater impact on user experience than deletion errors (i.e., missed identification, failing to identify the target sound event). Some deletion errors have little impact on user experience, but even a small number of insertion errors can have serious consequences. Therefore, improving the accuracy of sound event detection models is crucial for enhancing the precision of sound event detection and improving user experience. Summary of the Invention

[0003] This section is provided to briefly introduce the concepts, which will be described in detail in the Detailed Description section later. This section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, this disclosure provides a method for generating a sound event detection model, comprising: acquiring a first sample audio and a first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio; extracting audio features of the first sample audio and inputting the audio features of the first sample audio into a sound event detection model to obtain a first detection result; determining a target loss of the sound event detection model based on the first annotation result and the first detection result, wherein the target loss includes cross-entropy loss and continuity error penalty loss, the continuity error penalty loss being positively correlated with the number of audio frames with consecutive detection errors in the first detection result; and updating the model parameters of the sound event detection model based on the target loss.

[0005] Secondly, this disclosure provides a sound event detection method, comprising: acquiring an audio to be detected; extracting audio features of the audio to be detected, and inputting the audio features of the audio to be detected into a pre-trained sound event detection model to obtain a second detection result, wherein the sound event detection model is generated according to the sound event detection model generation method provided in the first aspect of this disclosure; and determining the start and end times of various types of sound events in the audio to be detected based on the second detection result.

[0006] Thirdly, this disclosure provides a sound event detection model generation apparatus, comprising: a first acquisition module, configured to acquire a first sample audio and a first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio; a first detection module, configured to extract audio features of the first sample audio and input the audio features of the first sample audio into a sound event detection model to obtain a first detection result; a first determination module, configured to determine a target loss of the sound event detection model based on the first annotation result and the first detection result, wherein the target loss includes cross-entropy loss and continuous error penalty loss, the continuous error penalty loss being positively correlated with the number of audio frames with consecutive detection errors in the first detection result; and an update module, configured to update the model parameters of the sound event detection model based on the target loss.

[0007] Fourthly, this disclosure provides a sound event detection device, comprising: a second acquisition module for acquiring audio to be detected; a second detection module for extracting audio features of the audio to be detected and inputting the audio features of the audio to be detected into a pre-trained sound event detection model to obtain a second detection result, wherein the sound event detection model is generated according to the sound event detection model generation method provided in the first aspect of this disclosure; and a second determination module for determining the start and end times of various sound events in the audio to be detected based on the second detection result.

[0008] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the sound event detection model generation method provided in the first aspect of this disclosure or the steps of the sound event detection method provided in the second aspect of this disclosure.

[0009] In a sixth aspect, this disclosure provides an electronic device, comprising: a storage device having at least one computer program stored thereon; and at least one processing device for executing the at least one computer program in the storage device to implement the steps of the sound event detection model generation method provided in the first aspect of this disclosure or the steps of the sound event detection method provided in the second aspect of this disclosure.

[0010] In the above technical solution, since continuous detection errors are the main cause of misidentification, the continuous error penalty term is used as part of the loss function in the training process of the sound event detection model. The model trained by this method can maintain high accuracy while keeping the sound event recall rate from decreasing. That is, it can greatly reduce misidentifications that affect user experience without introducing more missed identifications, thereby improving the accuracy of sound event detection and user experience.

[0011] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0013] Figure 1 This is a flowchart illustrating a sound event detection model generation method according to an exemplary embodiment.

[0014] Figure 2 This is a flowchart illustrating a method for determining a continuous error penalty loss according to an exemplary embodiment.

[0015] Figure 3 This is a flowchart illustrating a sound event detection method according to an exemplary embodiment.

[0016] Figure 4 This is a block diagram illustrating a sound event detection model generation apparatus according to an exemplary embodiment.

[0017] Figure 5 This is a block diagram illustrating a sound event detection device according to an exemplary embodiment.

[0018] Figure 6 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0019] As discussed in the background section, most fields require extremely high accuracy from sound event detection models. For example, wildlife detection uses animal calls to track their movements. Data collected by wildlife monitoring systems can be used to analyze basic information such as species composition, distribution, population size, and behavior, providing crucial reference material for wildlife conservation management and resource utilization. Another example is intelligent audio captioning. Common audio captioning automatically transcribes audio into text. Further, sound event detection can be used to display non-transparent content, such as applause, as symbols within the captions, enriching the caption presentation. These applications demand extremely high accuracy from sound event detection models.

[0020] Currently, to obtain a high-accuracy sound event detection model, a larger decision threshold is typically used for sound event categories requiring high accuracy during the model prediction phase. However, while this approach improves the model's detection accuracy, it leads to a decrease in the model's sound event recall. Since simply increasing the decision threshold does not introduce additional information to the model, improving accuracy comes at the cost of reduced recall.

[0021] In view of this, the present disclosure provides a model generation method, a sound event detection method, an apparatus, a medium, and a device.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0024] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] All actions involving the acquisition of signals, information, or data in this disclosure are carried out in accordance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0030] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0031] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0033] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0034] Figure 1This is a flowchart illustrating a sound event detection model generation method according to an exemplary embodiment. Figure 1 As shown, the sound event detection model generation method may include the following steps S101 to S104.

[0035] In S101, the first sample audio and the first annotation result of the first sample audio are obtained.

[0036] In this disclosure, first sample audio can be obtained from a training set, which may consist of various open-source audio data, or the audio data in the training set may be audio recorded by a user through a terminal for different application scenarios, such as a mobile phone, tablet computer, or voice recorder. All actions involving the acquisition of signals, information, or data in this disclosure are performed in accordance with the relevant data protection laws and regulations of the country where the user is located, and with authorization from the owner of the relevant device.

[0037] The first annotation result may include the start and end times of various sound events in the first sample audio.

[0038] In S102, the audio features of the first sample audio are extracted and input into the sound event detection model to obtain the first detection result.

[0039] In this disclosure, the audio features of the first sample audio can be obtained by inputting the first sample audio into a pre-trained feature extractor. These audio features can be, for example, log-Mel filter bank features, Mel-scale Frequency Cepstral Coefficients (MFCC) features, Constant Q Transform (CQT) features, fundamental frequency features, etc.

[0040] In one implementation, the first detection result may include the predicted sound event category for each first audio frame in the first sample audio.

[0041] In another implementation, the first detection result may include the predicted probability that each first audio frame in the first sample audio belongs to a class of sound events.

[0042] For example, sound events are divided into three categories: sound events of category A, sound events of category B, and sound events of category C. Thus, the first detection result includes the predicted probability that each first audio frame belongs to sound event of category A, the predicted probability that each first audio frame belongs to sound event of category B, and the predicted probability that each first audio frame belongs to sound event of category C.

[0043] In S103, based on the first annotation result and the first detection result, the target loss of the sound event detection model is determined. The target loss includes cross-entropy loss and continuity error penalty loss.

[0044] To improve the accuracy of the sound event detection model's output, current methods use median filtering to smooth the results, filtering out spikes in the model's predictions and smoothing out inaccurate predictions. In practice, if the erroneous audio frames are scattered throughout the audio, median filtering can easily remove these spikes. However, for continuous prediction errors, median filtering is ineffective. Clearly, such continuous errors have a greater impact on the final sound event detection result and are a major cause of misidentification. Therefore, this disclosure proposes a continuous error penalty term to specifically address this type of continuous error.

[0045] The consecutive error penalty loss is positively correlated with the number of consecutively detected audio frames in the first detection result; that is, the more consecutively detected audio frames in the first detection result, the greater the consecutive error penalty loss. The number of consecutively detected audio frames is the number of consecutively predicted audio frames.

[0046] In S104, the model parameters of the sound event detection model are updated based on the target loss.

[0047] In this disclosure, during the training of the sound event detection model, the target loss and the Adaptive Moment Estimation (Adam) optimizer can be used to update the model parameters. For example, the model is trained for 50 epochs using the target function and the Adam optimizer (one epoch means that each sample audio in the training set is trained once). The cross-entropy loss function in the target loss measures the difference between the first detection result and the first annotation result to calculate the error. The Adam optimizer is a gradient-based optimization algorithm with fast convergence, high computational efficiency, and low memory consumption.

[0048] In the above technical solution, since continuous detection errors are the main cause of misidentification, the continuous error penalty term is used as part of the loss function in the training process of the sound event detection model. The model trained by this method can maintain high accuracy while keeping the sound event recall rate from decreasing. That is, it can greatly reduce misidentifications that affect user experience without introducing more missed identifications, thereby improving the accuracy of sound event detection and user experience.

[0049] The structure of the aforementioned sound event detection model is described in detail below. In one embodiment, the sound event detection model can be a neural network model, which may include a convolutional neural network and a recurrent neural network connected in series. The convolutional neural network may include four sequentially connected convolutional neural network blocks, and the recurrent neural network is connected to the last of the four sequentially connected convolutional neural network blocks. Each convolutional neural network block includes a convolutional layer, a batch normalization layer, and a pooling layer connected in series. For example, the kernel size of the convolutional layer is 3*3, the number of channels in the four sequentially connected convolutional neural network blocks are 64, 128, 256, and 512 respectively, and the pooling size of the pooling layer is 2*2. The recurrent neural network may be, for example, a unidirectional long short-term memory neural network (LSTM) or a bidirectional long short-term memory neural network (Bi-LSTM), etc., and the number of neurons in this recurrent neural network is 256.

[0050] The following details the specific method for determining the penalty loss for continuous errors. Specifically, when the first detection result includes the predicted probability of each first audio frame in the first sample audio belonging to various types of sound events, it can be determined by... Figure 2 The following steps are implemented as shown in S201 to S204:

[0051] In S201, for each first audio frame, the labeled sound event category of the first audio frame is determined based on the first labeling result.

[0052] Specifically, for each first audio frame, the playback period of the first audio frame can be determined first, and then, based on the first annotation result, the sound event category corresponding to the playback period of the first audio frame can be determined as the annotation sound event category of the first audio frame.

[0053] In S202, a unique hot code is determined to label the category of the sound event.

[0054] For example, sound events are divided into three categories: Class A, Class B, and Class C. If the labeled sound event category of the first audio frame is Class B, then the unique hotkey for the labeled sound event category of the first audio frame is "010"; if the labeled sound event category of the first audio frame is Class A, then the unique hotkey for the labeled sound event category of the first audio frame is "100"; if the labeled sound event category of the first audio frame is Class C, then the unique hotkey for the labeled sound event category of the first audio frame is "001".

[0055] In S203, for each type of sound event, the continuity error penalty weight for the first audio frame belonging to that type of sound event is determined based on the predicted probability that the first audio frame belongs to that type of sound event and the predicted probability that the adjacent audio frames of the first audio frame belong to that type of sound event.

[0056] In this disclosure, the continuity error penalty weight is positively correlated with the number of consecutively detected audio frames in the first audio frame and its adjacent audio frames. That is, the more consecutively detected audio frames in the first audio frame and its adjacent audio frames, the greater the continuity error penalty weight, which means a greater error penalty. The adjacent audio frames of the first audio frame include the n frames preceding the first audio frame and the m frames following the first audio frame, where n ≥ 0, m ≥ 0, and n + m ≥ 1, and n and m are preset values.

[0057] For example, the continuity error penalty weight for the first audio frame belonging to the sound event class can be determined by the following equation (1) based on the predicted probability that the first audio frame belongs to the sound event class and the predicted probability that the adjacent audio frames of the first audio frame belong to the sound event class:

[0058]

[0059] Where, ω ij The continuity error penalty weight for the i-th first audio frame in the first sample audio belonging to the j-th sound event is calculated as the sum of the continuity errors of the i-th first audio frame, the n frames preceding the i-th first audio frame, and the m frames following the i-th first audio frame, where i = 1, 2, 3, ..., N, j = 1, 2, 3, ..., M, N is the number of first audio frames in the first sample audio, and M is the preset number of sound event categories; y xj Let x be the predicted probability that the x-th first audio frame in the first sample audio belongs to the j-th type of sound event, x = in,i-n+1,…,i,…,i+m-1,i+m.

[0060] For example, if m = n = 2, then x = i-2, i-1, i, i+1, i+2, i = ω ij It is the sum of the continuity errors of the i-th first audio frame, i.e., the two frames before and after it.

[0061] In S204, the continuous error penalty loss is determined based on the first detection result, the penalty weight for each continuous error, and each one-hot code.

[0062] For example, the successive error penalty loss can be determined based on the first detection result, the penalty weight for each successive error, and each one-hot code, using the following equation (2):

[0063]

[0064] Where, losscf Punitive loss for continuous errors; y is the unique hot key for the labeled sound event category of the i-th first audio frame in the first sample audio; ij Let be the predicted probability that the i-th first audio frame in the first sample audio belongs to the j-th type of sound event.

[0065] The following section details the method for determining the cross-entropy loss. Specifically, the cross-entropy loss can be determined based on the one-hot code of the labeled sound event category for each first audio frame in the first sample audio and the first detection result.

[0066] For example, the cross-entropy loss can be determined by the following equation (3) based on the one-hot code of the labeled sound event category for each first audio frame in the first sample audio and the first detection result:

[0067]

[0068] Where, loss crossentropy This represents the cross-entropy loss.

[0069] Figure 3 This is a flowchart illustrating a sound event detection method according to an exemplary embodiment. Figure 3 As shown, the method may include the following steps S301 to S303.

[0070] In S301, the audio to be detected is obtained.

[0071] In this disclosure, the audio to be detected can be any audio to be used for sound event detection.

[0072] In S302, the audio features of the audio to be detected are extracted and input into the pre-trained sound event detection model to obtain the second detection result.

[0073] In this disclosure, the sound event detection model is generated according to the sound event detection model generation method provided in this disclosure.

[0074] Alternatively, the audio features of the audio to be detected can be obtained by inputting the audio to be detected into a pre-trained feature extractor. These audio features can be, for example, log-Mel filter bank features, MFCC features, CQT features, fundamental frequency features, etc.

[0075] In one implementation, the second detection result may include the predicted sound event category for each second audio frame in the audio to be detected.

[0076] In another implementation, the second detection result may include the predicted probability that each second audio frame in the audio to be detected belongs to a certain type of sound event.

[0077] In S303, based on the second detection result, the start and end times of various sound events in the audio to be detected are determined.

[0078] In the above technical solution, since continuous detection errors are the main cause of misidentification, the continuous error penalty term is used as part of the loss function in the training process of the sound event detection model. The model trained by this method can maintain high accuracy while keeping the sound event recall rate from decreasing. That is, it can greatly reduce misidentifications that affect user experience without introducing more missed identifications, thereby improving the accuracy of sound event detection and user experience.

[0079] The following is a detailed description of the specific implementation method for determining the start and end times of various sound events in the audio to be detected based on the second detection result, as described in section 303 above. Specifically, this can be implemented through various methods. In one implementation method, the second detection result includes the predicted sound event category of each second audio frame in the audio to be detected. In this case, the start and end times of various sound events in the audio to be detected can be determined directly based on the predicted sound event category of each second audio frame.

[0080] In another implementation, the second detection result may include the predicted probability that each second audio frame in the audio to be detected belongs to a certain type of sound event. In this case, for each second audio frame in the audio to be detected, the predicted sound event category of the second audio frame can be determined according to the predicted probability that the second audio frame belongs to a certain type of sound event. Then, the start and end times of each type of sound event in the audio to be detected can be determined according to the predicted sound event category of each second audio frame.

[0081] The following is a detailed description of the specific implementation method for determining the predicted sound event category of the second audio frame based on the predicted probabilities of the second audio frame belonging to various types of sound events. Specifically, this can be implemented through various methods. In one implementation method, the sound event category corresponding to the maximum value among the predicted probabilities of the second audio frame belonging to various types of sound events can be determined as the predicted sound event category of the second audio frame.

[0082] For example, sound events are divided into three categories: category A, category B, and category C. The predicted probability of the second audio frame a belonging to category A is 0.5, the predicted probability of it belonging to category B is 0.8, and the predicted probability of it belonging to category C is 0.2. It is evident that the predicted probability of the second audio frame a belonging to category B is the highest among all predicted probabilities. Therefore, the predicted sound event category for the second audio frame a is category B.

[0083] In another implementation, if the maximum value of the predicted probability of the second audio frame belonging to various types of sound events is greater than a preset threshold, then the sound event category corresponding to the maximum value is determined as the predicted sound event category of the second audio frame; if the maximum value is less than or equal to the preset threshold, then the second audio frame is determined to be a non-sound event (e.g., a silent frame).

[0084] The following is a detailed description of the specific implementation method for determining the start and end times of various sound events in the audio to be detected based on the predicted sound event categories of each second audio frame. Specifically, this can be achieved in various ways. In one implementation method, the start and end times of various sound events in the audio to be detected can be determined directly based on the predicted sound event categories of each second audio frame.

[0085] Specifically, the playback periods of adjacent second audio frames with the same predicted sound event category can be merged to obtain the start and end times of various sound events in the audio to be detected.

[0086] For example, the audio to be detected includes second audio frames a, b, c, d, e, f, and g, with playback periods of 0s–0.5s, 0.5s–1s, 1s–1.5s, 1.5s–2s, 2s–2.5s, 2.5s–3s, and 3s–3.5s, respectively. The predicted sound event categories for second audio frames a, f, and g are all Class A sound events, while the predicted sound events for second audio frames b, c, d, and e are all Class C sound events. It is evident that the predicted sound event categories for second audio frames f and g are the same and adjacent, so their playback periods can be merged. Similarly, the predicted sound event categories for second audio frames b, c, d, and e are the same and consecutive, so their playback periods can also be merged. Therefore, the audio to be detected includes Class A sound events and Class C sound events. The start and end times of Class C sound events are 0.5s to 2.5s, and the start and end times of Class A sound events are 0s to 0.5s and 2.5s to 3.5s. That is, in the audio to be detected, 0s to 0.5s is Class A sound event, 0.5s to 2.5s is Class C sound event, and 2.5s to 3.5s is Class A sound event.

[0087] In another implementation, the start and end times of various sound events in the audio to be detected can be determined by the following steps (1) to (4):

[0088] Step (1): For each type of sound event in the audio to be detected, determine the second audio frame belonging to that type of sound event from the audio to be detected based on the predicted sound event category of each second audio frame, and obtain the reference audio frame.

[0089] Step (2): Encode the audio to be detected based on the reference audio frame to obtain the encoded sequence.

[0090] In this sequence, each element corresponds one-to-one with the second audio frame in the audio to be detected. The length of the encoding sequence is N. In the encoding sequence, the element corresponding to the reference audio frame is 1, and the rest are 0.

[0091] Step (3): Perform median filtering on the encoded sequence.

[0092] Step (4): Determine the start and end times of the sound events of this type in the audio to be detected based on the encoded sequence obtained after median filtering.

[0093] Specifically, the second audio frame corresponding to the element "1" in the encoded sequence obtained after median filtering in the audio frame to be detected can be identified as the second audio frame belonging to this type of sound event. The playback time periods of adjacent second audio frames of this type of sound event can be merged to obtain the start and end times of this type of sound event in the audio to be detected.

[0094] Among them, the start and end times of the sound events determined by the above method may overlap. In this case, we can first determine the maximum value of the predicted probability of the second audio frame of the overlapping period belonging to each type of sound event, and determine the sound event category corresponding to the maximum value as the final sound event category of the second audio frame of the overlapping period. Based on this result, the start and end times of the sound events with overlapping periods are adjusted.

[0095] For example, the audio to be detected includes second audio frames a, b, c, d, e, f, and g, with playback periods of 0s–0.5s, 0.5s–1s, 1s–1.5s, 1.5s–2s, 2s–2.5s, 2.5s–3s, and 3s–3.5s, respectively. The predicted sound event categories for second audio frames a, d, f, and g are all Class A sound events, while the predicted sound events for second audio frames b, c, and e are all Class C sound events. The median filter window length is 3.

[0096] It is evident that the audio to be detected includes both Class A and Class C sound events. Specifically, for Class A sound events, the second audio frames belonging to this class in the audio to be detected include second audio frames a, d, f, and g. Therefore, the reference audio frames corresponding to Class A sound events include second audio frames a, d, f, and g. Encoding the audio to be detected based on these reference audio frames yields the encoded sequence "1001011". Performing a median filter with a window length of 3 on this encoded sequence results in the encoded sequence "1000111". Thus, the second audio frames belonging to Class A sound events in the audio to be detected include second audio frames a, e, f, and g. Therefore, the start and end times of Class A sound events in the audio to be detected are 0s–0.5s and 2s–3.5s, respectively.

[0097] Similarly, for Class C sound events, the second audio frames belonging to this Class C sound event in the audio to be detected include second audio frame b, second audio frame c, and second audio frame e. That is, the reference audio frames corresponding to the Class C sound event include second audio frame b, second audio frame c, and second audio frame e. The audio to be detected is encoded based on these reference audio frames, resulting in the encoded sequence "0110100". This encoded sequence is then subjected to median filtering with a window length of 3, resulting in the encoded sequence "0111001". Therefore, the second audio frames belonging to the Class C sound event in the audio to be detected include second audio frame b, second audio frame c, second audio frame d, and second audio frame g. Thus, the start and end times of the Class C sound events in the audio to be detected are 0.5s to 2.5s and 3s to 3.5s.

[0098] It can be seen that in the audio to be detected, the start and end times of Class A sound events and Class C sound events overlap for a period of 3s to 3.5s. Among them, the prediction probabilities of the second audio frame g belonging to Class A sound events, Class B sound events, and Class C sound events during this overlapping period of 3s to 3.5s are 0.8, 0.2, and 0.5, respectively. Therefore, the final sound event category of the second audio frame g is Class A sound event. At this time, the start and end times of Class C sound events can be adjusted, that is, the 3s to 3.5s period can be removed.

[0099] Therefore, the start and end times of Class C sound events in the audio to be detected are 0.5s to 2.5s and 3s to 3.5s, respectively.

[0100] If there is a conflict in the sound event categories of the second audio frame g, the sound event category corresponding to the maximum predicted probability of the second audio frame g belonging to each type of sound event can be determined as the final sound category of the second audio frame g.

[0101] Therefore, the start and end times of Class A sound events in the audio to be detected are 0s to 0.5s and 2s to 3.5s, while the start and end times of Class C sound events are 0.5s to 2.5s.

[0102] In the above implementation, median filtering is used for smoothing to filter out some predicted spikes and smooth out depressions that should have been predicted as target events but were not, thereby further improving the accuracy of sound event detection.

[0103] The following is a detailed description of the specific implementation method for median filtering of the encoded sequence in step (3) above. Specifically, it can be implemented in a variety of ways. In one implementation method, the median filtering window length corresponding to various sound events is the same, which can be a preset length. In this way, the median filtering of the encoded sequence can be performed according to the preset length.

[0104] In another implementation, the target window length corresponding to a certain type of sound event can be determined first based on a pre-established correspondence between sound event categories and window lengths; then, median filtering can be performed on the encoded sequence based on the target window length.

[0105] In this way, the window length of the median filter matches the current sound event category, which is more consistent with the distribution of audio frames in actual audio, thereby improving the accuracy of sound event detection.

[0106] The following details the specific method for establishing the correspondence between the aforementioned sound event categories and window lengths. Specifically, this correspondence can be established through the following steps (a) to (c):

[0107] Step (a): Obtain multiple second sample audios and the second annotation result for each second sample audio.

[0108] In this disclosure, second sample audio can be obtained from the aforementioned training set. The second annotation result includes the start and end times of various sound events in the second sample audio.

[0109] Step (b): For each second sample audio, determine the duration of each type of sound event in the second sample audio based on the second annotation result of the second sample audio.

[0110] Step (c): For each type of sound event, determine the average duration of the sound event of that type in each second sample audio, and determine the window duration corresponding to the sound event of that type based on the average duration.

[0111] In one implementation, the window duration corresponding to this type of sound event can be determined based on the ratio of the average duration to the preset audio frame duration, wherein the window duration is an odd number.

[0112] Specifically, the ratio of the average duration to the preset audio frame duration can be rounded up or down to obtain the target ratio. If the target ratio is odd, the target ratio is determined as the window duration corresponding to this type of sound event. If the target ratio is even, (target ratio + 1) or (target ratio - 1) is determined as the window duration corresponding to this type of sound event.

[0113] Figure 4 This is a block diagram illustrating a sound event detection model generation apparatus according to an exemplary embodiment. Figure 4 As shown, the sound event detection model generation device 400 includes:

[0114] The first acquisition module 401 is used to acquire a first sample audio and a first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio;

[0115] The first detection module 402 is used to extract the audio features of the first sample audio and input the audio features of the first sample audio into the sound event detection model to obtain the first detection result;

[0116] The first determining module 403 is used to determine the target loss of the sound event detection model based on the first annotation result and the first detection result, wherein the target loss includes cross-entropy loss and continuous error penalty loss, and the continuous error penalty loss is positively correlated with the number of audio frames with continuous detection errors in the first detection result;

[0117] The update module 404 is used to update the model parameters of the sound event detection model based on the target loss.

[0118] In the above technical solution, since continuous detection errors are the main cause of misidentification, the continuous error penalty term is used as part of the loss function in the training process of the sound event detection model. The trained model can maintain high accuracy while keeping the sound event recall rate from decreasing. That is, without introducing more missed identifications, it can greatly reduce misidentifications that affect user experience, thereby improving the accuracy of sound event detection and user experience.

[0119] Optionally, the first detection result includes the predicted probability that each first audio frame in the first sample audio belongs to a certain type of sound event;

[0120] The sound event detection model generation device 400 further includes:

[0121] The third determining module is used to determine the continuous error penalty loss;

[0122] The third determining module includes:

[0123] The first determining submodule is used to determine the labeled sound event category of each first audio frame based on the first labeling result.

[0124] The second determining submodule is used to determine the unique hot code of the labeled sound event category;

[0125] The third determining submodule is used to determine the continuity error penalty weight of the first audio frame belonging to the first audio frame for each type of sound event based on the predicted probability that the first audio frame belongs to the first audio frame and the predicted probability that the adjacent audio frames of the first audio frame belong to the first audio frame. The continuity error penalty weight is positively correlated with the number of audio frames with consecutive detection errors in the first audio frame and its adjacent audio frames.

[0126] The fourth determining submodule is used to determine the continuous error penalty loss based on the first detection result, the continuous error penalty weight for each of the above, and the one-hot code for each of the above.

[0127] Figure 5 This is a block diagram illustrating a sound event detection device according to an exemplary embodiment. Figure 5 As shown, the sound event detection device 500 includes:

[0128] The second acquisition module 501 is used to acquire the audio to be detected;

[0129] The second detection module 502 is used to extract the audio features of the audio to be detected and input the audio features of the audio to be detected into a pre-trained sound event detection model to obtain a second detection result. The sound event detection model is generated according to the sound event detection model generation method provided in this disclosure.

[0130] The second determining module 503 is used to determine the start and end times of various sound events in the audio to be detected based on the second detection result.

[0131] In the above technical solution, since continuous detection errors are the main cause of misidentification, the continuous error penalty term is used as part of the loss function in the training process of the sound event detection model. The trained model can maintain high accuracy while keeping the sound event recall rate from decreasing. That is, without introducing more missed identifications, it can greatly reduce misidentifications that affect user experience, thereby improving the accuracy of sound event detection and user experience.

[0132] Optionally, the second detection result includes the predicted probability that each second audio frame in the audio to be detected belongs to various types of sound events;

[0133] The second determining module 503 includes:

[0134] The fifth determination submodule is used to determine the predicted sound event category of each second audio frame based on the predicted probability of the second audio frame belonging to various types of sound events.

[0135] The sixth determining submodule is used to determine the start and end times of various sound events in the audio to be detected based on the predicted sound event category of each of the second audio frames.

[0136] Optionally, the sixth determining submodule includes:

[0137] The seventh determining submodule is used to determine, for each type of sound event in the audio to be detected, a second audio frame belonging to that type of sound event from the audio to be detected based on the predicted sound event category of each second audio frame, and obtain a reference audio frame;

[0138] The encoding submodule is used to encode the audio to be detected according to the reference audio frame to obtain an encoding sequence, wherein the elements in the encoding sequence correspond one-to-one with the second audio frame in the audio to be detected, and the elements in the encoding sequence corresponding to the reference audio frame are 1, and the remaining elements are 0;

[0139] The first filtering submodule is used to perform median filtering on the encoded sequence;

[0140] The eighth determination submodule is used to determine the start and end times of the sound events of this type in the audio to be detected based on the encoded sequence obtained after median filtering.

[0141] Optionally, the first filtering submodule includes:

[0142] The ninth determination submodule is used to determine the target window length corresponding to a certain type of sound event based on a pre-established correspondence between sound event categories and window lengths.

[0143] The second filtering submodule is used to perform median filtering on the encoded sequence according to the target window length.

[0144] Optionally, the correspondence is established by a correspondence establishment device, wherein the correspondence establishment device may include:

[0145] The third acquisition module is used to acquire multiple second sample audios and a second annotation result for each second sample audio, wherein the second annotation result includes the start and end times of various sound events in the second sample audio;

[0146] The fourth determining module is used to determine the duration of various sound events in each of the second sample audios based on the second annotation result of the second sample audio.

[0147] The fifth determining module is used to determine the average duration of each type of sound event in each second sample audio for each type of sound event; the sixth determining module is used to determine the window duration corresponding to the type of sound event based on the average duration.

[0148] It should be noted that the aforementioned correspondence establishment device can be integrated into the aforementioned sound event detection device 500, or it can be independent of the aforementioned sound event detection device 500. This disclosure does not make any specific limitations.

[0149] In addition, this disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the sound event detection model generation method or the steps of the sound event detection method provided in this disclosure.

[0150] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0151] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0152] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0153] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0154] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0155] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0156] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0157] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following actions: to acquire a first sample audio and a first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio; to extract audio features of the first sample audio and input the audio features of the first sample audio into a sound event detection model to obtain a first detection result; to determine a target loss of the sound event detection model based on the first annotation result and the first detection result, wherein the target loss includes cross-entropy loss and continuity error penalty loss, the continuity error penalty loss being positively correlated with the number of audio frames with consecutive detection errors in the first detection result; and to update the model parameters of the sound event detection model based on the target loss.

[0158] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire an audio to be detected; extract audio features of the audio to be detected and input the audio features of the audio to be detected into a pre-trained sound event detection model to obtain a second detection result, wherein the sound event detection model is generated according to the sound event detection model generation method provided in this disclosure; and determine the start and end times of various sound events in the audio to be detected based on the second detection result.

[0159] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the modules do not necessarily limit the module itself; for example, the first acquisition module can also be described as "a module for acquiring a first sample audio and a first annotation result of the first sample audio".

[0162] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0163] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0164] According to one or more embodiments of this disclosure, Example 1 provides a method for generating a sound event detection model, comprising: acquiring a first sample audio and a first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio; extracting audio features of the first sample audio and inputting the audio features of the first sample audio into a sound event detection model to obtain a first detection result; determining a target loss of the sound event detection model based on the first annotation result and the first detection result, wherein the target loss includes cross-entropy loss and continuity error penalty loss, the continuity error penalty loss being positively correlated with the number of audio frames with consecutive detection errors in the first detection result; and updating the model parameters of the sound event detection model based on the target loss.

[0165] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein the first detection result includes the predicted probability that each first audio frame in the first sample audio belongs to a class of sound events; the continuity error penalty loss is determined by: for each first audio frame, determining the labeled sound event category of the first audio frame according to the first labeling result; determining the one-hot code of the labeled sound event category; for each class of sound events, determining the continuity error penalty weight of the first audio frame belonging to the class of sound events according to the predicted probability that the first audio frame belongs to the class of sound events and the predicted probability that the adjacent audio frames of the first audio frame belong to the class of sound events, wherein the continuity error penalty weight is positively correlated with the number of audio frames with consecutive detection errors in the first audio frame and its adjacent audio frames; and determining the continuity error penalty loss according to the first detection result, each of the continuity error penalty weights, and each of the one-hot codes.

[0166] According to one or more embodiments of this disclosure, Example 3 provides a sound event detection method, including: acquiring an audio to be detected; extracting audio features of the audio to be detected and inputting the audio features of the audio to be detected into a pre-trained sound event detection model to obtain a second detection result, wherein the sound event detection model is generated according to the sound event detection model generation method described in Example 1 or 2; and determining the start and end times of various types of sound events in the audio to be detected based on the second detection result.

[0167] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 3, wherein the second detection result includes the predicted probability that each second audio frame in the audio to be detected belongs to a class of sound events; the step of determining the start and end times of the various sound events in the audio to be detected based on the second detection result includes: for each second audio frame, determining the predicted sound event category of the second audio frame based on the predicted probability that the second audio frame belongs to a class of sound events; and determining the start and end times of the various sound events in the audio to be detected based on the predicted sound event category of each second audio frame.

[0168] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 4, wherein determining the start and end times of various types of sound events in the audio to be detected based on the predicted sound event category of each second audio frame includes: for each type of sound event in the audio to be detected, determining a second audio frame belonging to that type of sound event from the audio to be detected based on the predicted sound event category of each second audio frame, to obtain a reference audio frame; encoding the audio to be detected based on the reference audio frame to obtain an encoding sequence, wherein the elements in the encoding sequence correspond one-to-one with the second audio frames in the audio to be detected, and the elements in the encoding sequence corresponding to the reference audio frame are 1, and the remaining elements are 0; performing median filtering on the encoding sequence; and determining the start and end times of that type of sound event in the audio to be detected based on the encoding sequence obtained after median filtering.

[0169] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 5, wherein median filtering of the encoded sequence includes: determining a target window length corresponding to a pre-established correspondence between sound event categories and window lengths; and performing median filtering on the encoded sequence according to the target window length.

[0170] According to one or more embodiments of this disclosure, Example 7 provides the method of Example 6, wherein the correspondence is established by: obtaining a plurality of second sample audios and a second annotation result for each second sample audio, wherein the second annotation result includes the start and end times of various types of sound events in the second sample audios; for each second sample audio, determining the duration of various types of sound events in the second sample audio based on the second annotation result of the second sample audio; for each type of sound event, determining the average duration of the duration of that type of sound event in each second sample audio; and determining the window duration corresponding to that type of sound event based on the average duration.

[0171] According to one or more embodiments of this disclosure, Example 8 provides a sound event detection model generation apparatus, comprising: a first acquisition module, configured to acquire a first sample audio and a first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio; a first detection module, configured to extract audio features of the first sample audio and input the audio features of the first sample audio into a sound event detection model to obtain a first detection result; a first determination module, configured to determine a target loss of the sound event detection model based on the first annotation result and the first detection result, wherein the target loss includes cross-entropy loss and continuity error penalty loss, the continuity error penalty loss being positively correlated with the number of audio frames with consecutive detection errors in the first detection result; and an update module, configured to update the model parameters of the sound event detection model based on the target loss.

[0172] According to one or more embodiments of this disclosure, Example 9 provides the apparatus of Example 8, wherein the first detection result includes the predicted probability that each first audio frame in the first sample audio belongs to a class of sound events; the sound event detection model generation apparatus further includes: a third determining module, configured to determine the continuity error penalty loss; wherein the third determining module includes: a first determining submodule, configured to determine the labeled sound event category of each first audio frame based on the first labeling result; a second determining submodule, configured to determine the one-hot code of the labeled sound event category; a third determining submodule, configured to determine the continuity error penalty weight of each class of sound events based on the predicted probability that the first audio frame belongs to the class of sound events and the predicted probability that the adjacent audio frames of the first audio frame belong to the class of sound events, wherein the continuity error penalty weight is positively correlated with the number of audio frames with continuous detection errors in the first audio frame and its adjacent audio frames; and a fourth determining submodule, configured to determine the continuity error penalty loss based on the first detection result, each of the continuity error penalty weights, and each of the one-hot codes.

[0173] According to one or more embodiments of this disclosure, Example 10 provides a sound event detection device, comprising: a second acquisition module for acquiring audio to be detected; a second detection module for extracting audio features of the audio to be detected and inputting the audio features of the audio to be detected into a pre-trained sound event detection model to obtain a second detection result, wherein the sound event detection model is generated according to the sound event detection model generation method described in Example 1 or 2; and a second determination module for determining the start and end times of various types of sound events in the audio to be detected based on the second detection result.

[0174] According to one or more embodiments of this disclosure, Example 11 provides the apparatus of Example 10, wherein the second detection result includes the predicted probability that each second audio frame in the audio to be detected belongs to a class of sound events; the second determining module includes: a fifth determining submodule, configured to determine the predicted sound event category of each second audio frame based on the predicted probability that the second audio frame belongs to a class of sound events; and a sixth determining submodule, configured to determine the start and end times of each class of sound events in the audio to be detected based on the predicted sound event category of each second audio frame.

[0175] According to one or more embodiments of this disclosure, Example 12 provides an apparatus of Example 11, wherein the sixth determining submodule includes: a seventh determining submodule, configured to, for each type of sound event in the audio to be detected, determine a second audio frame belonging to that type of sound event from the audio to be detected based on the predicted sound event category of each second audio frame, to obtain a reference audio frame; an encoding submodule, configured to encode the audio to be detected based on the reference audio frame to obtain an encoded sequence, wherein the elements in the encoded sequence correspond one-to-one with the second audio frames in the audio to be detected, and the elements in the encoded sequence corresponding to the reference audio frame are 1, and the remaining elements are 0; a first filtering submodule, configured to perform median filtering on the encoded sequence; and an eighth determining submodule, configured to determine the start and end times of that type of sound event in the audio to be detected based on the encoded sequence obtained after median filtering.

[0176] According to one or more embodiments of this disclosure, Example 13 provides an apparatus of Example 12, wherein the first filtering submodule includes: a ninth determining submodule, configured to determine a target window length corresponding to a pre-established correspondence between sound event categories and window lengths; and a second filtering submodule, configured to perform median filtering on the encoded sequence based on the target window length.

[0177] According to one or more embodiments of this disclosure, Example 14 provides the apparatus of Example 13, wherein the correspondence is established by a correspondence establishment device, wherein the correspondence establishment device may include: a third acquisition module, configured to acquire a plurality of second sample audios and a second annotation result for each second sample audio, wherein the second annotation result includes the start and end times of various types of sound events in the second sample audios; a fourth determination module, configured to determine the duration of various types of sound events in each second sample audio based on the second annotation result of the second sample audio; a fifth determination module, configured to determine the average duration of the duration of each type of sound event in each second sample audio for each type of sound event; and a sixth determination module, configured to determine the window duration corresponding to the type of sound event based on the average duration.

[0178] According to one or more embodiments of the present disclosure, Example 15 provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, constitutes the steps of the method described in any one of Examples 1-7.

[0179] According to one or more embodiments of the present disclosure, Example 16 provides an electronic device including: a storage device having at least one computer program stored thereon; and at least one processing device for executing the at least one computer program in the storage device to implement the steps of the method described in any one of Examples 1-7.

[0180] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0181] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0182] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for generating a sound event detection model, characterized in that, include: Obtain the first sample audio and the first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio; The audio features of the first sample audio are extracted and input into the sound event detection model to obtain the first detection result. The first detection result includes the predicted probability that each first audio frame in the first sample audio belongs to various types of sound events. Based on the first annotation result and the first detection result, the target loss of the sound event detection model is determined, wherein the target loss includes cross-entropy loss and continuous error penalty loss, and the continuous error penalty loss is positively correlated with the number of audio frames with continuous detection errors in the first detection result; The sound event detection model is updated with its parameters based on the target loss. The continuous error penalty loss is determined in the following way: For each of the first audio frames, the labeled sound event category of the first audio frame is determined based on the first labeling result; Determine the unique-hot code for the labeled sound event category; For each type of sound event, based on the predicted probability that the first audio frame belongs to that type of sound event and the predicted probability that the adjacent audio frames of the first audio frame belong to that type of sound event, the continuity error penalty weight for the first audio frame belonging to that type of sound event is determined, wherein the continuity error penalty weight is positively correlated with the number of audio frames with consecutive detection errors in the first audio frame and its adjacent audio frames. The continuity error penalty loss is determined based on the first detection result, the continuity error penalty weight for each of the above, and the one-hot code for each of the above.

2. A method for detecting sound events, characterized in that, include: Obtain the audio to be detected; The audio features of the audio to be detected are extracted and input into a pre-trained sound event detection model to obtain a second detection result. The sound event detection model is generated by the sound event detection model generation method according to claim 1. Based on the second detection result, the start and end times of various sound events in the audio to be detected are determined.

3. The method according to claim 2, characterized in that, The second detection result includes the predicted probability that each second audio frame in the audio to be detected belongs to various types of sound events; The step of determining the start and end times of various sound events in the audio to be detected based on the second detection result includes: For each second audio frame, the predicted sound event category of the second audio frame is determined based on the predicted probability of the second audio frame belonging to various types of sound events; Based on the predicted sound event category of each second audio frame, the start and end times of various sound events in the audio to be detected are determined.

4. The method according to claim 3, characterized in that, The step of determining the start and end times of various sound events in the audio to be detected based on the predicted sound event category of each second audio frame includes: For each type of sound event in the audio to be detected, based on the predicted sound event category of each second audio frame, a second audio frame belonging to that type of sound event is determined from the audio to be detected, and a reference audio frame is obtained. The audio to be detected is encoded according to the reference audio frame to obtain an encoding sequence, wherein the elements in the encoding sequence correspond one-to-one with the second audio frame in the audio to be detected, and the elements in the encoding sequence corresponding to the reference audio frame are 1, and the remaining elements are 0; Median filtering is applied to the encoded sequence; Based on the encoded sequence obtained after median filtering, the start and end times of the sound events of this type in the audio to be detected are determined.

5. The method according to claim 4, characterized in that, The median filtering of the encoded sequence includes: Based on the pre-established correspondence between sound event categories and window lengths, determine the target window length corresponding to that type of sound event; Median filtering is performed on the encoded sequence based on the target window length.

6. The method according to claim 5, characterized in that, The correspondence is established in the following way: Obtain multiple second sample audios and a second annotation result for each second sample audio, wherein the second annotation result includes the start and end times of various sound events in the second sample audio; For each second sample audio, the duration of various sound events in the second sample audio is determined based on the second annotation result of the second sample audio; For each type of sound event, determine the average duration of that type of sound event in each second sample audio; based on the average duration, determine the window duration corresponding to that type of sound event.

7. A sound event detection model generation device, characterized in that, include: The first acquisition module is used to acquire a first sample audio and a first annotation result of the first sample audio, wherein the first annotation result includes the start and end times of various sound events in the first sample audio; The first detection module is used to extract the audio features of the first sample audio and input the audio features of the first sample audio into the sound event detection model to obtain the first detection result. The first detection result includes the predicted probability that each first audio frame in the first sample audio belongs to various types of sound events. The first determining module is used to determine the target loss of the sound event detection model based on the first annotation result and the first detection result, wherein the target loss includes cross-entropy loss and continuous error penalty loss, and the continuous error penalty loss is positively correlated with the number of audio frames with continuous detection errors in the first detection result; The update module is used to update the model parameters of the sound event detection model based on the target loss; The third determining module is used to determine the continuous error penalty loss; The third determining module includes: The first determining submodule is used to determine the labeled sound event category of each first audio frame based on the first labeling result. The second determining submodule is used to determine the unique hot code of the labeled sound event category; The third determining submodule is used to determine the continuity error penalty weight of the first audio frame belonging to the first audio frame for each type of sound event based on the predicted probability that the first audio frame belongs to the first audio frame and the predicted probability that the adjacent audio frames of the first audio frame belong to the first audio frame. The continuity error penalty weight is positively correlated with the number of audio frames with consecutive detection errors in the first audio frame and its adjacent audio frames. The fourth determining submodule is used to determine the continuous error penalty loss based on the first detection result, the continuous error penalty weight for each of the above, and the one-hot code for each of the above.

8. A sound event detection device, characterized in that, include: The second acquisition module is used to acquire the audio to be detected; The second detection module is used to extract the audio features of the audio to be detected and input the audio features of the audio to be detected into a pre-trained sound event detection model to obtain a second detection result, wherein the sound event detection model is generated by the sound event detection model generation method according to claim 1; The second determining module is used to determine the start and end times of various sound events in the audio to be detected based on the second detection result.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processing device, the program implements the steps of the method according to any one of claims 1-6.

10. An electronic device, characterized in that, include: A storage device having at least one computer program stored thereon; At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the method according to any one of claims 1-6.

Images