Artificial intelligence-based sound event detection method, apparatus, device, and medium

Abstract

The present application relates to the technical field of digital medical treatment, and in particular to a sound event detection method, device and equipment based on artificial intelligence and a medium. The method separates mixed sound into independent sound, inputs the independent sound into an encoder to obtain sound features, inputs the sound features into a recurrent layer to obtain timing features, uses a label prediction model to process and predict pseudo-event labels according to the sound features and the timing features, queries the pseudo-event labels as reference labels, uses the reference labels and the independent sound to form training samples, trains an event detection model, and then obtains an event detection result, extracts timing information of the sound features, enriches input of event prediction, improves accuracy of event prediction, queries and filters the pseudo-event labels to determine the reference labels, so that the event detection model better adapts to a scene, improves accuracy of sound event detection, and can assist medical staff in timely discovering abnormal sound events of patients in a medical environment, so as to respond in time.

Description

Technical Field

[0001] This invention relates to the field of digital medical technology, and in particular to a method, apparatus, device, and medium for detecting sound events based on artificial intelligence. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technology, sound event detection tasks based on artificial intelligence models have been widely applied in digital medical platforms. These platforms can support functions such as disease-assisted diagnosis, health management, and remote consultation, thereby improving the efficiency of medical institutions and facilitating residents' access to medical care.

[0003] Existing technologies typically employ deep convolutional models to perform sound event detection tasks. Sound event detection can be used to identify events from each sound source in complex multi-sound source scenarios. It can detect whether there are sounds matching the target event from the mixed sounds in the application scenario. For example, sound event detection can extract sound events such as patients groaning from a noisy ward environment, thus facilitating timely responses by medical staff.

[0004] However, deep convolutional models require a large amount of labeled data during training, and labeling mixed sounds is extremely difficult. The lack of training data leads to low accuracy of deep convolutional models in sound event detection. Moreover, even with sufficient training data, in real-time applications, the model struggles to effectively identify unlabeled sound events from mixed sounds, resulting in false positives and low accuracy. Therefore, improving the accuracy of sound event detection is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a sound event detection method, apparatus, device and medium based on artificial intelligence to solve the problem of low accuracy in sound event detection.

[0006] In a first aspect, embodiments of the present invention provide an artificial intelligence-based sound event detection method, the sound event detection method comprising:

[0007] The acquired mixed sound is input into the trained sound separation model to separate the sound according to the preset number of sound sources, resulting in N independent sounds, where N is an integer greater than zero;

[0008] For each independent sound, the independent sound is input into the encoder for feature extraction to obtain sound features. The sound features are input into the recurrent layer for temporal information extraction to obtain temporal features. The sound features and the temporal features are concatenated. The label prediction model is used to predict the label of the concatenated result to obtain the pseudo-event label corresponding to the independent sound. The N independent speech are traversed to obtain N pseudo-event labels.

[0009] Query each pseudo-event label in the preset target label set, determine the queried pseudo-event label as the reference label, and obtain M reference labels. Combine each reference label and its corresponding independent sound to form training samples, and obtain M training samples, where M is an integer greater than zero and less than N.

[0010] The encoder is fully connected to form an event detection model. The event detection model is trained based on the M training samples to obtain a trained event detection model. The N independent speech samples are input into the trained event detection model to perform event prediction. The predicted event categories of the N corresponding independent sounds are determined as the event detection results of the mixed sound.

[0011] Secondly, embodiments of the present invention provide an artificial intelligence-based sound event detection device, the sound event detection device comprising:

[0012] The sound separation module is used to input the acquired mixed sound into the trained sound separation model to separate the sound according to the preset number of sound sources, resulting in N independent sounds, where N is an integer greater than zero;

[0013] The label prediction module is used to extract features from each independent sound by inputting the independent sound into the encoder to obtain sound features, inputting the sound features into the recurrent layer to extract temporal information to obtain temporal features, concatenating the sound features and the temporal features, using the label prediction model to predict the label of the concatenated result to obtain the pseudo-event label corresponding to the independent sound, and traversing the N independent speech to obtain N pseudo-event labels.

[0014] The tag query module is used to query each pseudo-event tag in a preset target tag set, determine the queried pseudo-event tag as a reference tag, obtain M reference tags, and form training samples with each reference tag and its corresponding independent sound to obtain M training samples, where M is an integer greater than zero and less than N.

[0015] The event detection module is used to perform fully connected processing on the encoder to form an event detection model, train the event detection model based on the M training samples to obtain a trained event detection model, input the N independent speech into the trained event detection model to perform event prediction, and determine the predicted event categories of the N corresponding independent sounds as the event detection results of the mixed sound.

[0016] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sound event detection method as described in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the sound event detection method as described in the first aspect.

[0018] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0019] The acquired mixed sound is input into a trained sound separation model to separate the sound into N independent sounds according to a preset number of sound sources. For each independent sound, the sound is input into an encoder for feature extraction to obtain sound features. The sound features are then input into a recurrent layer for temporal information extraction to obtain temporal features. The sound features and temporal features are concatenated, and a label prediction model is used to predict the label of the concatenated result to obtain pseudo-event labels for the corresponding independent sounds. This process is repeated for N independent speech sounds to obtain N pseudo-event labels. Each pseudo-event label is queried in a preset target label set, and the queried pseudo-event label is determined as a reference label, resulting in M ​​reference labels. Each reference label and its corresponding independent sound are combined to form training samples, resulting in M ​​training samples. The encoder is then subjected to fully connected processing to form an event detection model. The event detection model is trained using M training samples to obtain a trained event detection model. N independent speech samples are then input into the trained event detection model for event prediction. The predicted event categories of the N corresponding independent sounds are determined as the event detection results for mixed sounds. Temporal information of sound features is extracted through a recurrent layer, enriching the input information for event label prediction and thus improving the accuracy of event label prediction. Furthermore, pseudo-event labels are queried and filtered to determine reference labels, enabling the event detection model trained based on these reference labels to better adapt to mixed sound scenarios. This avoids false detections due to interference from irrelevant labels, thereby improving the accuracy of sound event detection. This can assist medical personnel in promptly identifying abnormal sound events in patients within a medical environment, allowing for timely responses. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of an application environment for an artificial intelligence-based sound event detection method provided in Embodiment 1 of the present invention;

[0022] Figure 2 This is a flowchart illustrating an artificial intelligence-based sound event detection method provided in Embodiment 1 of the present invention;

[0023] Figure 3 This is a flowchart illustrating an artificial intelligence-based sound event detection method provided in Embodiment 2 of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of an artificial intelligence-based sound event detection device provided in Embodiment 3 of the present invention;

[0025] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0036] The first embodiment of this invention provides an artificial intelligence-based sound event detection method, which can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] The client and server can be deployed in a digital healthcare platform. The digital healthcare platform can support functions such as disease-assisted diagnosis, health management, and remote consultation. The sound event detection task is used to assist medical staff in the digital healthcare platform to perform timely tasks such as disease-assisted diagnosis, health management, and remote consultation. For example, it can detect the patient's groans in a noisy environment to assist in disease diagnosis, or detect the patient's consultation response information in a complex environment to help improve the accuracy of remote consultation.

[0038] See Figure 2 This is a flowchart illustrating an artificial intelligence-based sound event detection method provided in Embodiment 1 of the present invention. The sound event detection method described above can be applied to... Figure 1 The client, a computer device corresponding to the client, connects to the server to obtain a mixed sound and a set of target labels from the server. The mixed sound can refer to the sound that needs to be detected for sound events. The set of target labels can include at least one preset target label. The computer device corresponding to the client is equipped with a trained sound separation model, an encoder, and a recurrent layer. The trained sound separation model can be used to separate the mixed sound into multiple independent sounds. The encoder can be used to extract the sound features of the independent sounds, and the recurrent layer can be used to extract the temporal information of the sound features. Figure 2 As shown, the sound event detection method may include the following steps:

[0039] Step S201: Input the acquired mixed sound into the trained sound separation model to separate the sound according to the preset number of sound sources, and obtain N independent sounds.

[0040] Here, mixed sound can refer to sound collected in a multi-source scenario. In a medical scenario, a multi-source scenario can refer to a ward scenario, a consultation interaction scenario, etc. For example, in a ward scenario, it includes sound emitted by medical equipment as a sound source, sound emitted by the environment as a sound source, sound emitted by irrelevant personnel as a sound source, etc. The trained sound separation can be used to separate the mixed sound into independent sounds according to the sound source. An independent sound can refer to the sound emitted by a single sound source. The preset number of sound sources can be represented by N, where N is an integer greater than zero.

[0041] Specifically, the preset number of sound sources is 4 in this embodiment, that is, N is 4. By default, based on the prior condition that the mixed sound contains the sound emitted by 4 sound sources, the sound separation model separates the mixed sound into 4 independent sounds, and one independent sound corresponds to one sound source.

[0042] The training process of the sound separation model can use mixed sound samples as training samples. The mixed sound samples are obtained by mixing N initial independent sound samples. The mixed sound samples are input into the sound separation model to obtain N predicted independent sound samples. The metric distance between the N predicted independent sound samples and the N initial independent sound samples is calculated. The metric distance is used as the prediction loss. Based on the prediction loss, the parameters of the sound separation model are trained to obtain the trained sound separation model.

[0043] However, since the predicted independent sound samples obtained by the sound separation model are not ordered during sound separation, when using the labels of N predicted independent sound samples and mixed sound samples for distance measurement, it cannot be guaranteed that one predicted independent sound sample corresponds to one initial independent sound sample. The labels of the mixed sound samples are the N initial independent sound samples. Therefore, in this embodiment, all possible pairing methods of the N predicted independent sound samples and N initial independent sound samples are obtained, resulting in N! pairing methods, where N! can refer to the factorial of N. Distance measurement is performed for each pairing method, and the minimum value among the obtained distance measurements is determined as the prediction loss. In this embodiment, the distance measurement can be calculated using Euclidean distance.

[0044] The above steps involve inputting the acquired mixed sound into a trained sound separation model to separate the sound into N independent sounds according to a preset number of sound sources. By using the sound separation model to process the mixed sound into independent sounds, it is easier to predict the event information corresponding to each independent sound in the subsequent process. This avoids the problem of mutual interference between independent sounds during sound event detection, which leads to a decrease in the accuracy of sound event detection, and improves the accuracy of sound event detection.

[0045] Step S202: For each independent sound, input the independent sound into the encoder for feature extraction to obtain sound features, input the sound features into the recurrent layer for temporal information extraction to obtain temporal features, concatenate the sound features and temporal features, use the label prediction model to predict the label of the concatenation result to obtain the pseudo-event label of the corresponding independent sound, traverse N independent speech to obtain N pseudo-event labels.

[0046] The encoder can be used to extract feature information of independent sounds, sound features can be used to represent feature information of independent sounds, recurrent layers can be used to extract temporal information from sound features, temporal features can be used to represent temporal information from sound features, concatenation can be done by methods such as concatenation and feature value addition, the label prediction model can be used to predict the label to which an independent sound belongs based on sound features and temporal features, and pseudo-event labels can refer to the labels to which an independent sound belongs as predicted by the label prediction model.

[0047] Specifically, the loop layer can be implemented using gated loop units. In this embodiment, the sound features are input into two gated loop units connected in series for feature extraction, so as to fully extract the temporal features of the sound features.

[0048] Optionally, the encoder includes convolutional layers and attention layers;

[0049] Independent sounds are input into the encoder for feature extraction, resulting in sound features including:

[0050] Independent sounds are input into a convolutional layer to extract features according to time frames, resulting in frame-level features.

[0051] Frame-level features are input into the attention layer for feature aggregation, and the result of feature aggregation is determined to be audio features.

[0052] The convolutional layer can be used to extract deep features of independent sounds, namely frame-level features. One frame-level feature corresponds to one time frame. The channel dimension of the frame-level feature includes M channels, and one channel corresponds to one time frame. A time frame can refer to the smallest unit of frame division of independent sounds. In this embodiment, one time frame is 0.25 seconds. The attention layer can be used to extract the self-attention information of the frame-level features and play the role of feature aggregation.

[0053] This embodiment extracts self-attention information of frame-level features through an attention layer, thereby further extracting effective features based on deep features, improving the representation ability of sound features, and thus improving the accuracy of subsequent sound event prediction.

[0054] Optionally, the attention layer includes a global attention layer and a local attention layer;

[0055] Frame-level features are input into the attention layer for feature aggregation, and the resulting audio features are determined as follows:

[0056] The temporal dimension of the frame-level features is reduced by using a preset convolution kernel to obtain dimensionality-reduced features with a dimension of K.

[0057] The dimensionality-reduced features are input into the global attention layer to model the global context, resulting in global features with a dimension of K.

[0058] The global features are input into the local attention layer to model the local context, resulting in K local features.

[0059] The K local features are input into the global attention layer for global context modeling again, and the feature aggregation result is obtained. The feature aggregation result is determined to be the sound feature.

[0060] The preset convolution kernel can refer to a 1*1 size convolution kernel, which can be used for feature dimensionality reduction to achieve feature aggregation. The time dimension of the frame-level feature is also the channel dimension mentioned above. The initial channel dimension of the frame-level feature is M, and after dimensionality reduction, the channel dimension is K, where K is an integer greater than zero and less than M.

[0061] The global attention layer can be used to extract global attention information between all channel dimensions of the dimensionality reduction feature. The global feature can be used to represent this global attention information. The local attention layer can be used to extract local attention information of the global feature. One channel corresponds to one local feature, so there are a total of K local features.

[0062] The K local features are then aggregated again using a global attention layer to obtain the feature aggregation result as the sound feature.

[0063] In this embodiment, a multi-granularity attention mechanism is used to aggregate the dimensionality-reduced features, fully mining the self-attention information within the dimensionality-reduced features, thereby providing richer and more representative sound features for subsequent event prediction, and improving the accuracy of sound event prediction.

[0064] Optionally, global features can be input into a local attention layer for local context modeling, resulting in K local features, including:

[0065] For any dimension of the sub-feature in the global features, determine the corresponding preset window based on the dimension of the sub-feature;

[0066] After multiplying the global features with the preset window, the result is input into the local attention layer for local context modeling, which yields the local features of the corresponding sub-features. By traversing all sub-features, K local features are obtained.

[0067] In this embodiment, the global feature and the dimensionality reduction feature have the same size, that is, the channel dimension of the global feature is K. The global feature includes K sub-features corresponding to the channel dimension. The preset window can be used to control the interaction object of the sub-feature during feature interaction. The interaction object can refer to other sub-features. In this embodiment, the size of the preset window is 3, that is, the sub-feature only interacts with its adjacent sub-features. Multiplying the global feature and the preset window can mean setting the feature values ​​other than the preset window to 0, so as to achieve the purpose of local interaction.

[0068] This embodiment uses a preset window to control the interaction objects of sub-features, thereby conveniently realizing local feature interaction and improving the efficiency of local attention information extraction.

[0069] The above steps, for each independent sound, involve inputting the independent sound into an encoder for feature extraction to obtain sound features, inputting the sound features into a recurrent layer for temporal information extraction to obtain temporal features, concatenating the sound features and temporal features, and using a label prediction model to predict the label of the concatenated result to obtain the pseudo-event label for the corresponding independent sound. This process is repeated for N independent speech samples to obtain N pseudo-event labels. By extracting additional temporal information from the sound features through the recurrent layer, the input information of the label prediction model is enriched, thereby improving the accuracy of pseudo-event label prediction.

[0070] Step S203: Query each pseudo-event label in the preset target label set, determine the queried pseudo-event label as the reference label, obtain M reference labels, and combine each reference label and its corresponding independent sound to form training samples, thus obtaining M training samples.

[0071] The target label set can refer to the target event labels in the mixed sound scene known under prior conditions. For example, in the device monitoring scene, the target event labels can include device alarm, device lag, etc. The reference labels can be used to form training samples, and the training samples can be used to train the encoder in the future. M is an integer greater than zero and less than N.

[0072] Specifically, in order to ensure generalization ability, the label prediction model usually does not only predict for a single scene. As a result, after processing the mixed sound, the pseudo-event labels obtained are not only for the scene to which the mixed sound belongs. In this case, in order to improve the adaptability of the subsequent event detection model in the scene to which the mixed sound belongs, the pseudo-event labels are filtered so that the pseudo-event labels of the scene to which the mixed sound belongs can be used as the training labels of the subsequent event detection model.

[0073] The above steps involve querying each pseudo-event label in a preset target label set, determining the queried pseudo-event label as a reference label, obtaining M reference labels, and then using each reference label and its corresponding independent sound to form training samples to obtain M training samples. Through the pseudo-event label filtering process, the training samples of the subsequent event detection model are made more consistent with the scene to which the mixed sound belongs, and irrelevant labels are isolated from the interference of model training, thereby improving the event detection accuracy of the event detection model.

[0074] Step S204: Perform fully connected processing on the encoder to form an event detection model. Train the event detection model with M training samples to obtain a trained event detection model. Input N independent speech into the trained event detection model to predict events. Determine the predicted event category of the N corresponding independent sounds as the event detection result of mixed sound.

[0075] Among them, fully connected processing can refer to connecting a fully connected layer after the encoder. The fully connected layer can be used for classification tasks.

[0076] Optionally, the event detection model can be trained using M training samples to obtain a trained event detection model, including:

[0077] Input the individual sounds from each training sample into the event detection model to obtain the initial predicted category for each individual sound;

[0078] Based on the initial predicted category and reference label corresponding to each independent sound, the prediction sub-loss is calculated, and the sum of all prediction sub-losses is determined as the prediction loss. The event detection model is trained based on the prediction loss to obtain the trained event detection model.

[0079] The initial prediction category can refer to the prediction result of the independent sound input to the event detection model, and the prediction sub-loss can refer to the difference quantification value between the initial prediction category and the reference label corresponding to the independent sound. In this embodiment, the prediction sub-loss can be calculated using the cross-entropy loss function.

[0080] In this embodiment, the event detection model is trained simultaneously with M training samples. Under the premise that the M training samples belong to the same application scenario, the adaptability of the trained event detection model to the application scenario is improved, thereby improving the accuracy of sound event detection.

[0081] Optionally, by performing event prediction on an event detection model trained with N independent speech inputs, the predicted event categories of the N corresponding independent sounds are determined to be mixed sound event detection results, including:

[0082] The event detection model is trained with N independent speech inputs to predict the event, resulting in N predicted event categories corresponding to the independent sounds.

[0083] Statistical analysis is performed on the predicted event categories of N corresponding independent sounds, and at least one non-repeating predicted event category is obtained as the event detection result of the mixed sound.

[0084] In this embodiment, since the N independent speech segments contained in the mixed sound may correspond to multiple events, after obtaining the predicted event categories corresponding to the N independent speech segments, all predicted event categories are statistically analyzed to obtain non-repeating predicted event categories as the event detection results.

[0085] The above steps involve performing fully connected processing on the encoder to form an event detection model, training the event detection model with M training samples to obtain a trained event detection model, inputting N independent speech samples into the trained event detection model for event prediction, and determining the predicted event category of the N corresponding independent sounds as the event detection result of mixed sound. By training the event detection model with training samples adapted to the scene, and then inputting the mixed sound of the scene into the trained event detection model, the adaptability of the event detection model to the scene can be effectively improved, thereby increasing the accuracy of sound event detection.

[0086] This embodiment enables the simultaneous acquisition of multiple event detection results for sound events, avoiding missed detections and thus improving the accuracy of sound event detection.

[0087] This embodiment extracts temporal information of sound features through a recurrent layer, enriching the input information for event label prediction and thus improving the accuracy of event label prediction. Furthermore, it queries and filters pseudo-event labels to determine reference labels, enabling the event detection model trained based on the reference labels to better adapt to mixed sound scenarios. This avoids false detections of sound events due to interference from irrelevant labels, thereby improving the accuracy of sound event detection. It can assist medical staff in promptly detecting abnormal sound events of patients in the medical environment and responding accordingly.

[0088] See Figure 3 This is a flowchart illustrating an artificial intelligence-based sound event detection method provided in Embodiment 2 of the present invention. In this sound event detection method, the sound separation model can be trained by permutation traversal or by sample stage mixing.

[0089] When training using a permutation traversal method, the training method is described in Example 1 and will not be repeated here.

[0090] When training using a sample-stage mixing approach, the training process of the sound separation model includes the following steps:

[0091] Step S301: Obtain a first mixed sound sample and a second mixed sound sample, and mix the first mixed sound sample and the second mixed sound sample to obtain the overall mixed sound sample;

[0092] Step S302: Input the overall mixed sound sample into the sound separation model, and separate the sound according to the number of sound sources to obtain N independent sound samples;

[0093] Step S303: Multiply the learnable first matrix with N independent sound samples to obtain the first reconstructed sample; multiply the learnable second matrix with N independent sound samples to obtain the second reconstructed sample.

[0094] Step S304: Calculate the first reconstruction loss based on the first mixed sound sample, the first reconstructed sample, and the preset reconstruction loss function; calculate the second reconstruction loss based on the second mixed sound sample, the second reconstructed sample, and the reconstruction loss function.

[0095] Step S305: Based on the first reconstruction loss and the second reconstruction loss, train the sound separation model, the first matrix and the second matrix, and retain only the trained sound separation model.

[0096] The first mixed sound sample can be obtained by mixing a portion of the independent sound samples from N initial independent sound samples, and the second mixed sound sample can be obtained by mixing another portion of the independent sound samples from N initial independent sound samples. It should be noted that in this embodiment, both the first mixed sound sample and the second mixed sound sample are obtained by mixing at least two independent sound samples, and there is no overlap between the independent sound samples corresponding to the first mixed sound sample and the second mixed sound sample.

[0097] The overall mixed sound sample can refer to the result of mixing the first mixed sound sample and the second mixed sound sample, which is essentially still the result of mixing N initial independent sound samples.

[0098] The first matrix can be used to fuse N predicted independent sound samples. The first reconstructed sample can refer to the fusion result of the N independent sound samples through the first matrix. The second matrix can also be used to fuse N predicted independent sound samples. The second reconstructed sample can refer to the fusion result of the N independent sound samples through the second matrix.

[0099] The reconstruction loss function can be a metric function such as Euclidean distance or cosine similarity to obtain the difference between the mixed sound sample and the reconstructed sample. For example, in this embodiment, Euclidean distance is used as the reconstruction loss function. The first reconstruction loss can refer to the quantified difference between the first mixed sound sample and the first reconstructed sample, and the second reconstruction loss can refer to the quantified difference between the second mixed sound sample and the second reconstructed sample.

[0100] Specifically, only when the sound separation model can effectively separate each independent sound sample can the mixing result obtained based on the first matrix and the second matrix be sufficiently similar to the difference between the first mixed sound sample and the second mixed sound sample. Therefore, by using a two-stage method to mix N initial independent sound samples and using the mixing result of the first stage as a label, the separation effect of the sound separation model can be effectively supervised. The first stage of mixing can refer to mixing the N initial independent sound samples into a first mixed sound sample and a second mixed sound sample, and the second stage of mixing can refer to mixing the first mixed sound sample and the second mixed sound sample into a whole mixed sound sample.

[0101] Moreover, the role of the first matrix and the second matrix can be to learn the mixing information of the first mixed sample and the second mixed sample, that is, to learn which initial independent sound samples the first mixed sample is based on, without having to consider the arrangement of the output independent sound samples, as the information of their arrangement has already been learned by the first matrix and the second matrix.

[0102] After obtaining the first reconstruction loss and the second reconstruction loss, the first reconstruction loss and the second reconstruction loss are added together, and the sum is used as the reconstruction loss for training. Based on the reconstruction loss, the sound separation model, the first matrix and the second matrix are trained to obtain the trained sound separation model, the trained first matrix and the trained second matrix. At this time, only the trained sound separation model needs to be retained, because in actual use, the trained sound separation model only needs to complete the task of sound separation, without considering whether the separated sounds conform to a certain arrangement order. Only during the training process, in order to facilitate the calculation of reconstruction loss, it is necessary to learn the arrangement of the initial independent sound samples so as to calculate the difference between the initial independent sound samples and the corresponding independent sound samples.

[0103] In this embodiment, a two-stage hybrid approach is used to construct the input samples for the sound separation model. The hybrid result of the first stage is used as the label. Additionally, a learnable first matrix and a second matrix are added after the sound separation model to learn the arrangement information of independent sound samples. Compared with conventional training methods, this approach can greatly reduce the amount of computation during the training process of the sound separation model while ensuring its accuracy, thereby improving the training efficiency of the sound separation model.

[0104] Corresponding to the AI-based sound event detection method in the above embodiments, Figure 4 A structural block diagram of an AI-based sound event detection device according to Embodiment 3 of the present invention is shown. This sound event detection device is applied to a client, and the computer device corresponding to the client is connected to a server to obtain a mixed sound and a target tag set from the server. The mixed sound can refer to the sound for which sound event detection is required, and the target tag set can include at least one preset target tag. The computer device corresponding to the client is equipped with a trained sound separation model, an encoder, and a loop layer. The trained sound separation model can be used to separate the mixed sound into multiple independent sounds, the encoder can be used to extract the sound features of the independent sounds, and the loop layer can be used to extract the temporal information of the sound features. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0105] See Figure 4 The sound event detection device includes:

[0106] The sound separation module 41 is used to input the acquired mixed sound into the trained sound separation model to separate the sound according to the preset number of sound sources, and obtain N independent sounds, where N is an integer greater than zero;

[0107] The label prediction module 42 is used to extract features from the independent sound by inputting the independent sound into the encoder to obtain sound features, inputting the sound features into the recurrent layer to extract temporal information to obtain temporal features, concatenating the sound features and temporal features, and using the label prediction model to predict the label of the concatenated result to obtain the pseudo-event label of the corresponding independent sound. Iterates through N independent speech to obtain N pseudo-event labels.

[0108] The tag query module 43 is used to query each pseudo-event tag in the preset target tag set, determine the queried pseudo-event tag as a reference tag, obtain M reference tags, and form training samples with each reference tag and its corresponding independent sound to obtain M training samples, where M is an integer greater than zero and less than N.

[0109] The event detection module 44 is used to perform fully connected processing on the encoder to form an event detection model. The event detection model is trained based on M training samples to obtain a trained event detection model. N independent speech inputs are used to predict events in the trained event detection model, and the predicted event categories of the N corresponding independent sounds are determined as the event detection results of mixed sound.

[0110] Optionally, the aforementioned sound event detection device further includes:

[0111] The sample mixing module is used to acquire a first mixed sound sample and a second mixed sound sample, and to mix the first mixed sound sample and the second mixed sound sample to obtain the overall mixed sound sample;

[0112] The sample separation module is used to input the overall mixed sound sample into the sound separation model, and separate the sound according to the number of sound sources to obtain N independent sound samples;

[0113] The sample reconstruction module is used to multiply a learnable first matrix with N independent sound samples to obtain a first reconstructed sample, and to multiply a learnable second matrix with N independent sound samples to obtain a second reconstructed sample;

[0114] The loss calculation module is used to calculate the first reconstruction loss based on the first mixed sound sample, the first reconstructed sample and the preset reconstruction loss function, and to calculate the second reconstruction loss based on the second mixed sound sample, the second reconstructed sample and the reconstruction loss function.

[0115] The model training module is used to train the sound separation model, the first matrix, and the second matrix based on the first reconstruction loss and the second reconstruction loss, and only retains the trained sound separation model.

[0116] Optionally, the encoder includes convolutional layers and attention layers;

[0117] The aforementioned label prediction module 42 includes:

[0118] The feature extraction submodule is used to input independent sounds into the convolutional layer and extract features according to time frames to obtain frame-level features;

[0119] The feature aggregation submodule is used to input frame-level features into the attention layer for feature aggregation, and determine the feature aggregation result as audio features.

[0120] Optionally, the attention layer includes a global attention layer and a local attention layer;

[0121] The aforementioned feature aggregation submodule includes:

[0122] The feature dimensionality reduction unit is used to reduce the temporal dimension of the frame-level features using a preset convolution kernel to obtain dimensionality-reduced features. The dimension of the dimensionality-reduced features is K, where K is an integer greater than zero.

[0123] The first global modeling unit is used to input the dimensionality-reduced features into the global attention layer to perform global context modeling and obtain global features with a dimension of K.

[0124] The local modeling unit is used to input global features into the local attention layer to perform local context modeling, resulting in K local features;

[0125] The second global modeling unit is used to input K local features into the global attention layer to perform global context modeling again, obtain the feature aggregation result, and determine the feature aggregation result as sound features.

[0126] Optionally, the aforementioned local modeling unit includes:

[0127] The window determines the sub-unit, which is used to determine the corresponding preset window based on the dimension of the sub-feature for any dimension of the global features;

[0128] The feature traversal sub-unit is used to multiply the global features and the preset window, and then input them into the local attention layer for local context modeling to obtain the local features of the corresponding sub-features. By traversing all sub-features, K local features are obtained.

[0129] Optionally, the event detection module 44 mentioned above includes:

[0130] The initial prediction submodule is used to input the independent sounds from each training sample into the event detection model to obtain the initial predicted category for each independent sound.

[0131] The prediction loss calculation submodule is used to calculate the prediction sub-loss based on the initial prediction category and reference label corresponding to each independent sound, determine the sum of all prediction sub-losses as the prediction loss, and train the event detection model based on the prediction loss to obtain the trained event detection model.

[0132] Optionally, the event detection module 44 mentioned above includes:

[0133] The category prediction submodule is used to perform event prediction on the event detection model trained with N independent speech inputs to obtain N predicted event categories corresponding to independent sounds.

[0134] The category statistics submodule is used to perform statistics on the predicted event categories of N corresponding independent sounds, and obtain at least one non-repeating predicted event category as the event detection result of the mixed sound.

[0135] It should be noted that the information interaction and execution process between the above modules, sub-modules, units and sub-units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and the resulting technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0136] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor, wherein the processor executes the computer program to implement the steps in any of the above-described embodiments of the sound event detection method.

[0137] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0138] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0139] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0140] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0141] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0142] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0144] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An artificial intelligence-based sound event detection method, characterized by, The sound event detection method comprises: The acquired mixed sound input is input into a trained sound separation model to separate sounds according to a preset number of sound sources, and N independent sounds are obtained, N being an integer greater than zero; For each independent sound, the independent sound is input into an encoder for feature extraction to obtain a sound feature, the sound feature is input into a recurrent layer for time sequence information extraction to obtain a time sequence feature, the sound feature and the time sequence feature are spliced, a label prediction model is used to predict the spliced result to obtain a pseudo event label corresponding to the independent sound, and the N independent sounds are traversed to obtain N pseudo event labels; Each pseudo event label is queried in a preset target label set, the queried pseudo event label is determined as a reference label, M reference labels are obtained, each reference label and the independent sound corresponding thereto form a training sample, and M training samples are obtained, M being an integer greater than zero and less than N; The encoder is fully connected to form an event detection model, the M training samples are used to train the event detection model, and a trained event detection model is obtained, the N independent sounds are input into the trained event detection model for event prediction, and a prediction event category of the N independent sounds obtained is determined as an event detection result of the mixed sound.

2. The voice event detection method of claim 1, wherein, The sound separation model training process comprises: A first mixed sound sample and a second mixed sound sample are acquired, the first mixed sound sample and the second mixed sound sample are mixed to obtain an overall mixed sound sample; The overall mixed sound sample is input into the sound separation model, and sound separation is performed according to the number of sound sources to obtain N independent sound samples; A first matrix is multiplied by the N independent sound samples to obtain a first reconstruction sample, and a second matrix is multiplied by the N independent sound samples to obtain a second reconstruction sample; A first reconstruction loss is calculated according to the first mixed sound sample, the first reconstruction sample and a preset reconstruction loss function, and a second reconstruction loss is calculated according to the second mixed sound sample, the second reconstruction sample and the reconstruction loss function; The sound separation model, the first matrix and the second matrix are trained based on the first reconstruction loss and the second reconstruction loss, and only the trained sound separation model is retained.

3. The voice event detection method of claim 1, wherein, The encoder comprises a convolution layer and an attention layer; The independent sound is input into the encoder for feature extraction to obtain a sound feature, which comprises: The independent sound is input into the convolution layer for feature extraction according to a time frame to obtain a frame-level feature; The frame-level feature is input into the attention layer for feature aggregation, and the feature aggregation result is determined as the sound feature.

4. The sound event detection method of claim 3, wherein, The attention layer comprises a global attention layer and a local attention layer; The frame-level feature is input into the attention layer for feature aggregation, and the feature aggregation result is determined as the sound feature, which comprises: The time dimension of the frame-level feature is processed by a preset convolution kernel to obtain a dimension-reduced feature, the dimension of the dimension-reduced feature is K, and K is an integer greater than zero. ​ input the dimension-reduced feature into the global attention layer for global context modeling to obtain a global feature, wherein a dimension of the global feature is K; input the global feature into the local attention layer for local context modeling to obtain K local features; input the K local features into the global attention layer again for global context modeling to obtain the feature aggregation result, and determine the feature aggregation result as the sound feature.

5. The sound event detection method of claim 4, wherein, The inputting the global feature into the local attention layer for local context modeling to obtain K local features comprises: for a sub-feature in any dimension of the global feature, determining a preset window corresponding to the sub-feature according to the dimension of the sub-feature; multiplying the global feature and the preset window, and inputting the multiplication result into the local attention layer for local context modeling to obtain a local feature corresponding to the sub-feature, and traversing all sub-features to obtain the K local features.

6. The voice event detection method of claim 1, wherein, The training of the event detection model according to the M training samples comprises: inputting each independent sound in each training sample into the event detection model to obtain an initial prediction category corresponding to each independent sound; calculating a prediction sub-loss according to the initial prediction category corresponding to each independent sound and a reference label, determining a sum of all prediction sub-losses as a prediction loss, and training the event detection model based on the prediction loss to obtain the trained event detection model.

7. The method of any one of claims 1 to 6, wherein, The inputting the N independent sounds into the trained event detection model for event prediction to determine a prediction event category of each corresponding independent sound as the event detection result of the mixed sound comprises: inputting the N independent sounds into the trained event detection model for event prediction to obtain a prediction event category of each corresponding independent sound; counting the prediction event categories of the N corresponding independent sounds to obtain at least one non-repeated prediction event category as the event detection result of the mixed sound.

8. An artificial intelligence-based sound event detection apparatus, characterized by comprising: The sound event detection device comprises: a sound separation module configured to input the obtained mixed sound into a trained sound separation model for sound separation according to a preset number of sound sources to obtain N independent sounds, wherein N is an integer greater than zero; a label prediction module configured to input each independent sound into an encoder for feature extraction to obtain a sound feature, input the sound feature into a recurrent layer for timing information extraction to obtain a timing feature, concatenate the sound feature and the timing feature, use a label prediction model to predict a label of the concatenation result to obtain a pseudo event label corresponding to the independent sound, and traverse the N independent sounds to obtain N pseudo event labels; a label query module configured to query each pseudo event label in a preset target label set, determine a queried pseudo event label as a reference label, obtain M reference labels, and group each reference label and its corresponding independent sound to form a training sample to obtain M training samples, wherein M is an integer greater than zero and less than N. An event detection module is configured to fully connect the encoder to form an event detection model, train the event detection model according to the M training samples, obtain a trained event detection model, input the N independent sounds into the trained event detection model to perform event prediction, and determine a prediction event category of the N independent sounds as an event detection result of the mixed sound.

9. A computer device, comprising: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the sound event detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executable on the processor to implement the sound event detection method according to any one of claims 1 to 7.

Images