Audio event identification method and system, electronic equipment and storage medium

Abstract

The invention discloses an audio event recognition method and system, electronic equipment and a storage medium, and relates to the technical field of computer artificial intelligence. The method comprises the following steps: performing spatial domain filtering on a collected initial audio signal through a signal preprocessing module to obtain a filtered to-be-recognized target audio signal; performing physical feature extraction and data enhancement on the target audio signal based on a target audio extraction physical rule through a physical feature processing module to obtain an enhanced target physical feature; performing feature coding fusion according to the target physical feature and the target audio signal through a feature coding fusion module to obtain a fusion feature; and performing feature recognition on the fused features through a feature recognition output module to obtain an initial audio event recognition result, performing correction compensation and classified output processing based on the initial audio event recognition result, and outputting a target audio event recognition result. Based on the improved physical constraint neural network model, the adaptability and the recognition precision during audio event recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of computer artificial intelligence technology, and in particular to an audio event recognition method and system, electronic device and storage medium. Background Technology

[0002] With the development of science and technology, audio event recognition technology has been widely applied in intelligent products and services across various industries. In recent years, audio event recognition methods based on deep learning and neural networks have become the most advantageous technical solutions in this field. Currently, neural network-based audio event recognition technology relies on traditional feature representation methods such as MFCC / Mel spectrograms. While these features perform well in speech recognition, their frequency domain resolution and temporal structure representation capabilities are limited, making it difficult to effectively capture the significantly different acoustic characteristics of different types of sound events. It is also difficult to simultaneously highlight the acoustic characteristics of different types of sounds. For example, sudden noises (such as glass breaking, impact sounds, and bird calls) have obvious transient characteristics and non-stationary harmonic structures, while bird calls have complex harmonic modes and frequency modulation characteristics. Traditional feature representation methods lack generalization ability in these scenarios, leading to decreased model discrimination and recognition accuracy. Furthermore, current audio event recognition technology uses a single-microphone system, resulting in low recognition accuracy in complex scenarios (such as factories and streets) with multiple sound source aliasing interference. How to achieve a highly adaptable and accurate audio event recognition method is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes an audio event recognition method, system, electronic device, and storage medium, which can improve the adaptability and recognition accuracy of audio event recognition based on an improved physically constrained neural network model.

[0004] In a first aspect, embodiments of this application provide an audio event recognition method applied to an audio event recognition system, the audio event recognition system comprising: a signal preprocessing module and a physical constraint neural network model connected in sequence, wherein the physical constraint neural network model comprises: a physical feature processing module, a feature encoding fusion module, and a feature recognition output module connected in sequence; The method includes: The signal preprocessing module performs spatial filtering on the initial audio signal to obtain the filtered target audio signal to be identified. The physical feature processing module performs physical feature extraction and data enhancement processing on the target audio signal based on the matching target audio extraction physical rules to obtain the enhanced target physical features. The feature encoding fusion module performs feature encoding fusion processing based on the target physical features and the target audio signal to obtain fused features. The feature recognition output module performs feature recognition processing on the fused features to obtain an initial audio event recognition result. Based on the initial audio event recognition result, it performs correction and compensation processing and classification output processing to output the target audio event recognition result.

[0005] Secondly, embodiments of this application provide an audio event recognition system, including: a signal preprocessing module and a physical constraint neural network model connected in sequence, wherein the physical constraint neural network model includes: a physical feature processing module, a feature encoding fusion module, and a feature recognition output module connected in sequence; the signal preprocessing module, the physical feature processing module, the feature encoding fusion module, and the feature recognition output module cooperate with each other to enable the audio event recognition system to implement the audio event recognition method as described in any one of the embodiments of the first aspect.

[0006] Thirdly, embodiments of this application provide an electronic device, including at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the audio event recognition method as described in any of the embodiments of the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the audio event recognition method as described in any of the embodiments of the first aspect.

[0008] The embodiments of this application include: The audio event recognition system includes a signal preprocessing module and a physical constraint neural network model connected in sequence. The physical constraint neural network model includes a physical feature processing module, a feature encoding and fusion module, and a feature recognition output module connected in sequence. In the process of audio event recognition using the system, firstly, the signal preprocessing module performs spatial filtering on the acquired initial audio signal to obtain the filtered target audio signal to be recognized. Adding spatial filtering in the front-end processing of the audio event recognition system enhances the main sound source, providing a good foundation for recognition and facilitating subsequent improvements in recognition accuracy. Secondly, the physical feature processing module, based on the matched target sound... The target audio signal is subjected to physical feature extraction and data augmentation processing based on frequency extraction physical rules to obtain enhanced target physical features. Targeted data augmentation is performed on different types of target audio signals based on physical laws to improve adaptability and recognition accuracy during audio event recognition. Then, through the feature encoding fusion module, feature encoding fusion processing is performed on the target physical features and the target audio signal to obtain fused features. Finally, through the feature recognition output module, feature recognition processing is performed on the fused features to obtain an initial audio event recognition result. Based on the initial audio event recognition result, correction and compensation processing and classification output processing are performed to output the target audio event recognition result. High-precision recognition is ensured through feature encoding fusion processing, correction and compensation processing, and classification output processing. In other words, the embodiments of this application can improve the adaptability and recognition accuracy during audio event recognition based on an improved physical constraint neural network model.

[0009] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description and the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the system architecture of an audio event recognition system provided in one embodiment of this application; Figure 2 This is a schematic diagram of the specific architecture of an audio event recognition system provided in one embodiment of this application; Figure 3 This is a flowchart illustrating an audio event recognition method provided in one embodiment of this application; Figure 4 This is a schematic diagram of the overall process of an audio event recognition method provided in one embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0012] It should be noted that although a logical order is shown in the flowcharts in this application, in some cases, the steps shown or described may be performed in a different order than that shown in the flowcharts. In the description of this application, "several" means one or more, and "more" means two or more. The terms "first" and "second" are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order in which the technical features are indicated.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0014] First, let me explain some of the terms used in this application: Physically constrained neural network models: Neural network architectures that incorporate physical features into the neural network structure to constrain the neural network to not violate physical rules during the learning process.

[0015] Hilbert envelope gradient: It is a key feature used in signal processing to quantify the intensity of transient changes in a signal. Its calculation process combines Hilbert transform and gradient operation, and it is particularly suitable for detecting sudden transient components in audio events (such as impact sounds and breaking sounds).

[0016] Feature compression refers to a technique that preserves key information by reducing feature dimensions or the amount of data.

[0017] This application provides an audio event recognition method, an audio event recognition system, an electronic device, and a computer-readable storage medium, relating to the field of computer artificial intelligence technology. The method includes: spatially filtering the acquired initial audio signal using a signal preprocessing module to obtain a filtered target audio signal to be recognized; extracting physical features and enhancing data from the target audio signal based on physical rules for target audio extraction using a physical feature processing module to obtain enhanced target physical features; fusing features based on the target physical features and the target audio signal using a feature encoding and fusion module to obtain fused features; performing feature recognition on the fused features using a feature recognition output module to obtain an initial audio event recognition result; and performing correction, compensation, and classification output processing based on the initial audio event recognition result to output the target audio event recognition result. This method, based on an improved physically constrained neural network model, enhances the adaptability and recognition accuracy of audio event recognition.

[0018] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, the audio event recognition system 1000 includes a signal preprocessing module 100 and a physical constraint neural network model 200 connected in sequence. The physical constraint neural network model 200 includes a physical feature processing module 210, a feature encoding fusion module 220 and a feature recognition output module 230 connected in sequence.

[0020] The functions of the physical feature processing module 210, the feature encoding fusion module 220, and the feature recognition output module 230 are further explained as follows.

[0021] Specifically, the signal preprocessing module 100 is used to: perform spatial filtering on the acquired initial audio signal to obtain the filtered target audio signal to be identified; The physical feature processing module 210 is used to: perform physical feature extraction and data enhancement processing on the target audio signal based on the matching target audio extraction physical rules, so as to obtain the enhanced target physical features.

[0022] The feature encoding fusion module 220 is used to: perform feature encoding fusion processing based on the target physical features and the target audio signal to obtain fused features.

[0023] The feature recognition output module 230 is used to: perform feature recognition processing on the fused features to obtain the initial audio event recognition result, perform correction and compensation processing and classification output processing based on the initial audio event recognition result, and output the target audio event recognition result.

[0024] The specific structures of the physical feature processing module, feature encoding and fusion module, and feature recognition output module are further explained. For example... Figure 2As shown, specifically, the signal preprocessing module 100 includes a multi-microphone array 110 and a signal filtering module 120 connected in sequence; the physical feature processing module 210 includes a physical feature extraction module 211 and a data augmentation module 212 connected in sequence; the feature encoding and fusion module 220 includes a feature encoding module 221 and a feature fusion module 222 connected in sequence; wherein, the feature encoding module 221 includes a pre-constructed first neural network model 221A and a second neural network model 221B; the feature recognition output module 230 includes a physical constraint layer 231, a physical loss compensation module 232 and a classification output module 233 connected in sequence.

[0025] In the audio event recognition system 1000 provided in this application embodiment, the signal preprocessing module 100, the physical feature processing module 210, the feature encoding fusion module 220, and the feature recognition output module 230 cooperate with each other to enable the audio event recognition system to implement the audio event recognition method provided in this application embodiment. Based on the improved physical constraint neural network model, it can improve the adaptability and recognition accuracy when performing audio event recognition.

[0026] Those skilled in the art will understand that the system structure shown in the figures does not constitute a limitation on the embodiments of this application, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0027] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0028] It will be understood by those skilled in the art that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. It is known by those skilled in the art that with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0029] Based on the above system structure, various embodiments of the audio event recognition method of this application are proposed below.

[0030] like Figure 3 As shown, this application provides an audio event recognition method, applicable to, for example... Figure 1The audio event recognition system shown includes a signal preprocessing module and a physical constraint neural network model connected in sequence. The physical constraint neural network model further includes a physical feature processing module, a feature encoding and fusion module, and a feature recognition output module connected in sequence. This audio event recognition method may include, but is not limited to, steps S100 to S400.

[0031] Step S100: The initial audio signal is spatially filtered by the signal preprocessing module to obtain the filtered target audio signal to be identified.

[0032] Step S200: The physical feature processing module performs physical feature extraction and data enhancement processing on the target audio signal based on the matching target audio extraction physical rules to obtain the enhanced target physical features.

[0033] Step S300: The feature encoding fusion module performs feature encoding fusion processing based on the target physical features and the target audio signal to obtain the fused features.

[0034] Step S400: The feature recognition output module performs feature recognition processing on the fused features to obtain the initial audio event recognition result. Based on the initial audio event recognition result, the module performs correction and compensation processing and classification output processing to output the target audio event recognition result.

[0035] In this embodiment, through steps S100 to S400, during the audio event recognition process using an audio event recognition system, firstly, the initial audio signal is spatially filtered by a signal preprocessing module to obtain the filtered target audio signal to be recognized. Adding spatial filtering in the front-end processing of the audio event recognition system enhances the main sound source, providing a good foundation for recognition and facilitating subsequent improvements in recognition accuracy. Secondly, a physical feature processing module extracts physical features and enhances data on the target audio signal based on matching target audio extraction physical rules, obtaining enhanced target physical features. Targeted data enhancement is applied to different types of target audio signals based on physical laws to improve adaptability and recognition accuracy during audio event recognition. Then, a feature encoding fusion module performs feature encoding fusion processing on the target physical features and the target audio signal to obtain fused features. Finally, a feature recognition output module performs feature recognition processing on the fused features to obtain the initial audio event recognition result. Based on the initial audio event recognition result, correction and compensation processing and classification output processing are performed to output the target audio event recognition result. High-precision recognition is ensured through feature encoding fusion processing, correction and compensation processing, and classification output processing. In other words, the embodiments of this application can improve the adaptability and recognition accuracy when performing audio event recognition based on an improved physical constraint neural network model.

[0036] According to some embodiments of this application, the signal preprocessing module includes a multi-microphone array and a signal filtering module connected in sequence. Further explanation of step S100: Step S100 involves the signal preprocessing module performing spatial filtering on the acquired initial audio signal to obtain the filtered target audio signal to be identified, including but not limited to steps S110 to S120.

[0037] Step S110: Acquire the initial audio signal using a multi-microphone array.

[0038] Step S120: The initial audio signal is spatially filtered using a spatial filtering algorithm via a signal filtering module to obtain the target audio signal.

[0039] Understandably, a multi-microphone array is set up in the environment under test to acquire the initial audio signal in the environment under test.

[0040] Specifically, spatial filtering algorithms (especially beamforming) are algorithms that utilize spatial information to process signals. Spatial filtering algorithms achieve spatial selectivity, enhancing signals in the target direction and suppressing noise in the interfering direction; they are an indispensable key technology for solving multi-source interference and improving the signal-to-noise ratio in microphone array applications.

[0041] Current audio event recognition technologies primarily employ single-microphone systems. However, single-microphone systems are susceptible to interference from multiple sound sources, resulting in low recognition rates in complex scenarios (such as factories and streets) with overlapping sound sources. This application's embodiment, through steps S110 to S120, acquires signals based on a multi-microphone array and uses a spatial filtering algorithm to suppress noise from directions other than the primary sound source, thereby enhancing the signal strength from the primary sound source. In other words, this application enhances the primary sound source by adding spatial filtering in the front-end processing of the audio event recognition system, providing a solid foundation for recognition and facilitating subsequent improvements in recognition accuracy. According to some embodiments of this application, the physical feature processing module includes a physical feature extraction module and a data enhancement module connected in sequence. Further explanation of step S200: Step S200 involves the physical feature processing module performing physical feature extraction and data enhancement processing on the target audio signal based on the matched target audio extraction physical rules, to obtain enhanced target physical features, including but not limited to steps S210 to S230.

[0042] Step S210: The physical feature extraction module determines the matching target audio extraction physical rule from multiple candidate audio extraction physical rules in the preset physical rule library according to the type of the target audio signal; wherein, a candidate audio extraction physical rule is configured corresponding to a type of target audio signal.

[0043] Step S220: The physical feature extraction module determines the type of target parameter to be extracted based on the physical rules for target audio extraction, and extracts the corresponding initial physical features from the target audio signal according to the type of target parameter.

[0044] Step S230: The data augmentation module determines the target augmentation method based on the physical rules for extracting the target audio, and performs data augmentation processing on the initial physical features using the target augmentation method to obtain the enhanced target physical features.

[0045] Specifically, in step S210, the type of the target audio signal is simply identified. The type of the target audio signal includes, but is not limited to, human voices, bird calls, and audio in a specific frequency band. This application does not impose specific restrictions on the type of target audio signal.

[0046] Specifically, in step S210, the physical rule base includes multiple candidate audio extraction physical rules. Each candidate audio extraction physical rule is configured corresponding to a type of target audio signal. This application does not impose a specific limit on the number of candidate audio extraction physical rules. There is a unique mapping relationship between the candidate audio extraction physical rules and the type of audio signal. Therefore, the matching target audio extraction physical rule can be determined based on the type of the target audio signal to facilitate the subsequent extraction of initial physical features and data augmentation processing.

[0047] In step S220, the types of target parameters to be extracted include, but are not limited to: harmonic parameters (such as fundamental frequency, harmonic intensity, harmonic attenuation rate, etc.), transient event detection parameters (e.g., Hilbert envelope gradient), zero-crossing rate, duration, environmental acoustic parameters, and other physical features. The Hilbert envelope gradient, for sudden / transient events (such as glass breaking, impact sound), is characterized by a sharp rise followed by rapid or slow decay; the envelope has a very steep peak. The envelopes of continuous events (such as engine roar, wind noise) are relatively flat, without sharp peaks. The envelopes of periodic events (such as bird calls, bell sounds) exhibit regular, periodic fluctuations.

[0048] In step S230, when performing data enhancement on the physical features of different types of target audio signals, the target enhancement methods indicated by the physical rules for target audio extraction vary. Specifically, for target audio signals with harmonic structures, such as birds and human voices, the target enhancement methods used are harmonic masking, translation, or scaling; for target audio signals with specific frequency bands, the target enhancement method used is filtering enhancement; and for target audio signals with transient event background noise, the target enhancement method used is extended enhancement. Specifically, when performing data enhancement on the physical features of different types of target audio signals, corresponding filtering enhancement methods need to be used.

[0049] To illustrate steps S220 to S230, when the target audio signal is birdsong, a matching physical rule for extracting the target audio is determined from a preset physical rule library. This physical rule indicates that the target parameter to be extracted is a harmonic parameter and the filtering enhancement method is a filtering enhancement method. Based on this, the corresponding initial physical features (i.e., harmonic parameters) are extracted from the target audio signal. The extracted initial physical features (i.e., harmonic parameters) are then filtered and enhanced to obtain the enhanced target physical features.

[0050] In this embodiment of the application, steps S210 to S230 are used to perform targeted data enhancement on different types of target audio signals based on physical laws, so as to improve the adaptability and recognition accuracy when performing audio event recognition.

[0051] According to some embodiments of this application, the feature encoding fusion module includes: a feature encoding module and a feature fusion module connected in sequence; wherein, the feature encoding module includes: a pre-constructed first neural network model and a second neural network model. Further explanation of step S300: Step S300: The feature encoding fusion module performs feature encoding fusion processing based on the target physical features and the target audio signal to obtain fused features, including but not limited to steps S310 to S330.

[0052] Step S310: Using the first neural network model, perform physical encoding processing on the target physical features to obtain the first feature code.

[0053] Step S320: The target audio signal is processed by time-domain encoding using the second neural network model to obtain the second feature code.

[0054] Step S330: The first feature code and the second feature code are fused using the feature fusion module to obtain the fused features.

[0055] It should be noted that the first neural network model and the second neural network model are pre-trained and are processed simultaneously, but the specific processing procedures are different. The first neural network model is used to perform physical encoding on the input target physical features and output the first feature code. At the same time, the second neural network model is used to perform temporal encoding on the input target audio signal and output the second feature code.

[0056] Specifically, time-domain coding involves using methods such as wavelet transform, Mel-Frequency Cepstral Coefficients (MFCC), or Mel spectrum to encode the time-domain and frequency-domain features of the target audio signal. Wavelet transform is a mathematical tool used to analyze the time-frequency characteristics of signals (such as audio); Mel spectrum is a spectrum at the Mel scale, a frequency scale based on human auditory perception; MFCC is a set of compact feature vectors obtained through further processing of the Mel spectrum, which can efficiently compress information, capturing the most crucial spectral envelope characteristics of sound with a very small vector (e.g., 13-dimensional). This application does not impose specific restrictions on the processing methods used in time-domain coding.

[0057] According to some embodiments of this application, step S330 is further described. Step S330: The first feature code and the second feature code are subjected to feature fusion processing to obtain fused features, including but not limited to steps S331 to S332.

[0058] Step S331: Perform dimension alignment processing on the first feature code and the second feature code to obtain the first feature code and the second feature code with the same dimension.

[0059] Step S332: Perform feature merging on the first feature code and the second feature code with the same dimension to obtain the fused feature.

[0060] Specifically, the dimension alignment process in step S331 refers to mapping these two features with different shapes and meanings to the same vector space through mathematical operations, so that they have the same dimension; providing a data foundation for subsequent feature fusion.

[0061] It should be noted that this application does not impose specific restrictions on the dimensions of the first feature code and the second feature code obtained after the dimension alignment process in step S331, as long as the first feature code and the second feature code after dimension alignment are the same.

[0062] Specifically, step S332 merges and fuses the first feature encoding and the second feature encoding with the same dimension to obtain the fused feature. In this way, information from different sources is combined to generate a more powerful, robust and information-rich joint representation.

[0063] Steps S331 to S332 yielded richer fusion features, which helps improve the accuracy of subsequent identification.

[0064] According to some embodiments of this application, the feature recognition output module includes: a physical constraint layer, a physical loss compensation module, and a classification output module connected in sequence; after obtaining fused features by feature encoding fusion processing based on the target physical features and the target audio signal in step S300, the audio event recognition method further includes: determining target constraint conditions based on the type of the target audio signal through the physical constraint layer, performing constraint detection processing on the fused features based on the target constraint conditions, and obtaining detection results; when the detection results indicate that a transient event has been detected, feature compression operation is disabled in the feature recognition processing.

[0065] Specifically, this application sets up a physical constraint layer and constructs multiple constraint conditions based on the physical characteristics of different types of target audio signals to perform constraint detection processing and obtain detection results. When the detection result is: transient event is detected, feature compression operation is disabled in feature recognition processing to ensure signal integrity.

[0066] According to some embodiments of this application, step S400 is further described. Step S400: Based on the initial audio event recognition result, correction compensation processing and classification output processing are performed to output the target audio event recognition result, including but not limited to steps S410 to S420.

[0067] Step S410: The physical loss compensation module determines the target compensation method based on the target's physical characteristics, and corrects and compensates the initial audio event recognition result according to the target compensation method to obtain the compensated recognition result.

[0068] Step S420: The classification output module determines the category of the compensated recognition result and outputs the target audio event recognition result.

[0069] Further explanation of step S410: During the training phase of the physically constrained neural network model, a loss compensation mechanism is constructed based on the physical characteristics of different categories of target audio signals to prevent the physically constrained neural network model from going astray. For example, if harmonic structures are detected in bird audio, compensation is added to the model loss function to protect features from being lost. By comparing with the physical rule base, positive or negative loss compensation is applied to the rationality of the fundamental frequency of different categories of target audio signals. In the above loss compensation, different weight parameters can be assigned according to the importance of physical features.

[0070] Specifically, when utilizing the physical loss compensation module of the trained physical constraint neural network model, the target compensation method is determined based on the target physical characteristics as follows: First, it determines whether to perform positive or negative loss compensation based on the type of the target audio signal. After determining the initial compensation value, it determines the weight parameters corresponding to the physical characteristics of the target audio signal. The weight parameters are multiplied by the initial compensation value to determine the target compensation value. This target compensation value is then used to correct and compensate the initial audio event recognition result, resulting in a compensated recognition result that improves the accuracy of the audio event recognition. Finally, a classifier is used to classify the compensated recognition result and output the target audio event recognition result.

[0071] Steps S410 to S420 correct and compensate the initial audio event recognition results of different types of target audio signals, thereby increasing the accuracy and adaptability of the target audio event recognition results output by the physical constraint neural network model.

[0072] Combination Figure 4 Here is an example to illustrate the overall process of the audio event recognition method provided in the embodiments of this application.

[0073] Step S1: Acquire the initial audio signal using a multi-microphone array.

[0074] Step S2: Perform spatial filtering on the initial audio signal to obtain the target audio signal.

[0075] Step S3: Determine the physical rules for the target audio from the physical rule base.

[0076] Step S4: Extract physical rules based on the target audio, perform physical feature extraction and data augmentation based on physical constraints, and obtain the target physical features.

[0077] Step S5: Perform time-domain coding on the target audio signal to obtain the second feature code.

[0078] Step S6: Perform physical encoding processing on the target physical features to obtain the first feature code. Steps S5 and S6 are performed simultaneously.

[0079] Step S7: Fuse the first feature code and the second feature code.

[0080] Step S8: Perform feature recognition processing on the fused features to obtain the initial audio event recognition result, and then perform constraint detection processing on the physical constraint layer.

[0081] Step S9: Correct and compensate the initial audio event recognition result to obtain the compensated recognition result.

[0082] Step S10: Determine the category of the compensated recognition result and output the target audio event recognition result.

[0083] It should be noted that existing neural network-based audio event recognition technologies have the following problems: (1) Insufficient feature generalization: Traditional methods rely on MFCC / Mel spectrograms, which are difficult to take into account the acoustic features of different types of sounds, such as the harmonic structure and transient characteristics of sudden noises (such as glass breaking, impact sound, bird calls). (2) Multi-source interference: Single microphone systems have low recognition rates in complex scenes (such as factories, streets) with multi-source aliasing interference. (3) Model deviates from physical rules: Deep learning ignores the acoustic laws and characteristics of audio (such as fundamental frequency range, harmonic characteristics, reverberation time, etc.), resulting in a sharp drop in performance in low signal-to-noise ratio scenes. (4) Difficulty in taking into account audio data noise reduction: In general audio event recognition methods, noise reduction needs to retain the acoustic features of different types of audio, such as transient features and complex harmonic structures, which are difficult to take into account by traditional methods. (5) Poor interpretability of output results: The complex nonlinear transformations of neural network technology and the high-dimensional features of automatic learning are difficult to express in an understandable way. To address the above issues, this application enhances the main sound source by adding a multi-microphone array and spatial filtering in the audio front-end processing to overcome the problem of multi-source interference; physical feature extraction is added in the feature extraction stage to overcome the problem of the model deviating from physical rules; targeted data augmentation is performed on different categories of target audio signals based on physical laws to preserve the acoustic features of different categories of audio, thus overcoming the problem of difficulty in simultaneously reducing audio data noise; a physical constraint layer is added to ensure the integrity of audio features, and a physical loss compensation module corrects and compensates the output results of different categories of audio, increasing the controllability and interpretability of the model output results to overcome the problems of poor interpretability and insufficient feature generalization.

[0084] In summary, compared with other neural network audio event recognition methods, the audio event recognition method of this application embodiment is based on physical rules and performs data augmentation, making it more suitable for augmenting audio feature data of different categories; the audio features incorporate the physical rule characteristics of audio, improving the model's generalization ability; and the addition of a physical constraint layer and a physical loss compensation module during training increases the controllability and interpretability of the model's output results.

[0085] like Figure 5 As shown, this application also provides an electronic device, including: The processor 501 can be implemented using a general-purpose central processing unit, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory, static storage device, dynamic storage device, or random access memory. The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and called and executed by the processor 501 using the audio event recognition method of the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0086] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described audio event recognition method.

[0087] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the modules described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0089] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by this application.

Claims

1. An audio event recognition method, characterized in that, An audio event recognition system is applied to an audio event recognition system, which includes a signal preprocessing module and a physical constraint neural network model connected in sequence. The physical constraint neural network model includes a physical feature processing module, a feature encoding and fusion module, and a feature recognition output module connected in sequence. The method includes: The signal preprocessing module performs spatial filtering on the initial audio signal to obtain the filtered target audio signal to be identified. The physical feature processing module performs physical feature extraction and data enhancement processing on the target audio signal based on the matching target audio extraction physical rules to obtain the enhanced target physical features. The feature encoding fusion module performs feature encoding fusion processing based on the target physical features and the target audio signal to obtain fused features. The feature recognition output module performs feature recognition processing on the fused features to obtain an initial audio event recognition result. Based on the initial audio event recognition result, it performs correction and compensation processing and classification output processing to output the target audio event recognition result.

2. The audio event recognition method according to claim 1, characterized in that, The signal preprocessing module includes: a multi-microphone array and a signal filtering module connected in sequence; The step of performing spatial filtering on the acquired initial audio signal through the signal preprocessing module to obtain the filtered target audio signal to be identified includes: The initial audio signal is acquired through the multi-microphone array; The initial audio signal is spatially filtered using the signal filtering module based on a spatial filtering algorithm to obtain the target audio signal.

3. The audio event recognition method according to claim 1, characterized in that, The physical feature processing module includes: a physical feature extraction module and a data augmentation module connected in sequence; The physical feature processing module performs physical feature extraction and data enhancement processing on the target audio signal based on the matched target audio extraction physical rules to obtain enhanced target physical features, including: The physical feature extraction module determines a matching target audio extraction physical rule from multiple candidate audio extraction physical rules in a preset physical rule base, based on the type of the target audio signal; wherein, each candidate audio extraction physical rule is configured to correspond to a type of target audio signal. The physical feature extraction module determines the type of target parameter to be extracted based on the physical rules for target audio extraction, and extracts the corresponding initial physical features from the target audio signal according to the type of target parameter. The data enhancement module determines the target enhancement method based on the target audio extraction physical rules, and performs data enhancement processing on the initial physical features using the target enhancement method to obtain the enhanced target physical features.

4. The audio event recognition method according to claim 1, characterized in that, The feature encoding fusion module includes: a feature encoding module and a feature fusion module connected in sequence; wherein, the feature encoding module includes: a pre-constructed first neural network model and a second neural network model; The step of obtaining fused features by performing feature encoding and fusion processing based on the target physical features and the target audio signal through the feature encoding and fusion module includes: The first feature code is obtained by physically encoding the target physical features using the first neural network model. The target audio signal is processed by time-domain encoding using the second neural network model to obtain the second feature code; The feature fusion module performs feature fusion processing on the first feature code and the second feature code to obtain the fused feature.

5. The audio event recognition method according to claim 4, characterized in that, The step of performing feature fusion processing on the first feature code and the second feature code to obtain the fused feature includes: The first feature code and the second feature code are dimensionally aligned to obtain a first feature code and a second feature code with the same dimensions. The first feature encoding and the second feature encoding, which have the same dimension, are merged to obtain the fused feature.

6. The audio event recognition method according to claim 1, characterized in that, The feature recognition output module includes: a physical constraint layer, a physical loss compensation module, and a classification output module connected in sequence; after obtaining the fused features by performing feature encoding and fusion processing based on the target physical features and the target audio signal, the method further includes: Through the physical constraint layer, target constraint conditions are determined according to the type of the target audio signal, and constraint detection processing is performed on the fusion features according to the target constraint conditions to obtain the detection result; If the detection result indicates that a transient event has been detected, the feature compression operation is disabled in the feature recognition process.

7. The audio event recognition method according to claim 6, characterized in that, The step of performing correction and compensation processing and classification output processing based on the initial audio event recognition result, and outputting the target audio event recognition result, includes: The physical loss compensation module determines the target compensation method based on the target physical characteristics, and corrects and compensates the initial audio event recognition result according to the target compensation method to obtain the compensated recognition result. The classification output module performs category determination on the compensated recognition result and outputs the target audio event recognition result.

8. An audio event recognition system, characterized in that, include: The system comprises a signal preprocessing module and a physical constraint neural network model connected in sequence, wherein the physical constraint neural network model includes a physical feature processing module, a feature encoding fusion module, and a feature recognition output module connected in sequence; the signal preprocessing module, the physical feature processing module, the feature encoding fusion module, and the feature recognition output module cooperate with each other to enable the audio event recognition system to implement the audio event recognition method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, It includes at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the audio event recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the audio event recognition method as described in any one of claims 1 to 7.

Images