Multi-channel sound recognition and positioning method, device, equipment and storage medium
By masking the Gabor feature and GCC-PHAT feature of multi-channel sound, the problem of excessive coupling between parameters in the TCN model is solved, and the model generalization ability and recognition accuracy of sound event positioning and detection is improved.
Patent Information
- Application Number
- CN202210510713.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The coupling between parameters of the sound event positioning and detection model based on time domain convolutional network (TCN), resulting in weakening of model generalization ability and decreasing recognition positioning accuracy.
By obtaining the Gabor features and GCC-PHAT features of multi-channel sound, the GCC-PHAT features are masked using Gabor features, and the masked features are input into the network model, and the model is updated to reduce parameter coupling and improve the The generalization ability and recognition and positioning accuracy of the model.
It effectively improves the generalization ability and recognition and positioning accuracy of the model, reduces the amount of parameters, and reduces the memory requirements.
Smart Images

Figure CN115101062B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of sound recognition and positioning, and in particular to a method, device, equipment and storage medium for recognizing and positioning multi-channel sounds. Background Art
[0002] Emre from Finland proposed a convolutional recurrent neural network (CRNN) for sound event localization and detection (SELD), which first uses a convolutional neural network to extract recognition features, then uses a recurrent neural network to recognize events of complex sounds, and then separates and recognizes the included audio events, and uses two fully connected layers for positioning and recognition. However, SELD based on CRNN generates a large number of parameters, so using CRNN for sound event localization and detection requires higher memory bandwidth and a larger memory buffer.
[0003] In order to reduce the number of parameters generated in the process of sound event localization and detection, Karim Guirguis of the University of Stuttgart in Germany used causal convolution of the time domain convolutional network (TCN) in SELD. However, SELD based on TCN has the defect of too high coupling between parameters. The high coupling between parameters easily causes the weight optimization in the reverse process of model training to affect each other, which in turn leads to the weakening of the generalization ability of the model and the decrease of recognition and positioning accuracy. Summary of the invention
[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for identifying and locating multi-channel sounds, aiming to solve the technical problem in the prior art that the SELD based on TCN has too high coupling between parameters, which leads to the weakening of the generalization ability of the model and the decrease of recognition and positioning accuracy.
[0005] To achieve the above object, the present invention provides a method, device, equipment and storage medium for identifying and locating multi-channel sound, the method comprising the following steps:
[0006] Obtaining Gabor features and GCC-PHAT features of the first multi-channel sound, wherein the Gabor features include a first label of each audio segment in the first multi-channel sound, the first label is a real type, and the GCC-PHAT features include a second label of each audio segment in the first multi-channel sound, the label information is a real position and a real angle;
[0007] Inputting the Gabor features into a first model to obtain a predicted category of each audio segment in the first multi-channel sound;
[0008] Determine a mask of the GCC-PHAT feature according to the Gabor feature, and input the masked GCC-PHAT feature into a second model to obtain a predicted value of each audio segment in the first multi-channel sound, wherein the predicted value includes a predicted position and a predicted angle;
[0009] Obtain a target network model according to the first model and the second model;
[0010] The second multi-channel sound is input into the target network model to obtain the type, position and angle of the target audio segment in the second multi-channel sound.
[0011] Preferably, the obtaining of Gabor features of the first multi-channel sound includes:
[0012] Inputting the first basic features of the first multi-channel into four convolutional layers in the Gabor filter bank to obtain static features;
[0013] Inputting the static features into the differential operation layer in the Gabor filter bank to obtain dynamic features;
[0014] Based on the static feature and the dynamic feature, a Gabor feature of the first channel sound is determined.
[0015] Preferably, the step of inputting the first basic features of the first multi-channel into four convolutional layers in a Gabor filter bank to obtain static features includes:
[0016] Inputting the first basic features of the first multi-channels into the first convolutional layer of the four convolutional layers to obtain second basic features;
[0017] Inputting the second basic feature into the global mean pooling layer in the four convolutional layers to obtain a third basic feature;
[0018] Inputting the third basic feature into the Gaussian low-pass filter layer in the four convolutional layers to obtain a fourth basic feature;
[0019] The fourth basic feature is input into a normalization layer in the four convolutional layers to obtain a static feature.
[0020] Preferably, the obtaining of the GCC-PHAT feature of the first multi-channel sound includes:
[0021] Obtain a second label for each audio segment in the first multi-channel sound;
[0022] After determining the array response of each audio segment based on the second label, a GCC-PHAT feature of the first multi-channel sound is determined according to the array response.
[0023] Preferably, determining the mask of the GCC-PHAT feature according to the Gabor feature comprises:
[0024] Input the Gabor features into the sigmoid layer in the resnet18 network to generate random numbers;
[0025] The random number is used as a mask for the GCC-PHAT feature.
[0026] Preferably, obtaining a target network model according to the first model and the second model includes:
[0027] Obtaining a first error according to a difference between the first label and the predicted category;
[0028] Obtaining a second error according to a difference between the second label and the predicted value;
[0029] The first model and the second model are updated based on the first error and the second error to obtain a target network model.
[0030] Preferably, updating the first model and the second model based on the first error and the second error to obtain a target network model includes:
[0031] determining a joint loss based on the first error and the second error;
[0032] The first model and the second model are updated based on the joint loss to obtain a target network model.
[0033] In addition, to achieve the above-mentioned purpose, the present invention further proposes a multi-channel sound recognition and positioning device, the multi-channel sound recognition and positioning device comprising:
[0034] An acquisition module, used for acquiring Gabor features and GCC-PHAT features of the first multi-channel sound;
[0035] The acquisition module is further used to input the Gabor feature into the first model to obtain the predicted type of each audio segment in the first multi-channel sound;
[0036] The acquisition module is further used to determine a mask of the GCC-PHAT feature according to the Gabor feature, and input the masked GCC-PHAT feature into a second model to obtain a predicted value of each audio segment in the first multi-channel sound;
[0037] The acquisition module is further used to obtain a target network model according to the first model and the second model;
[0038] The identification and positioning module is used to input the second multi-channel sound into the target network model to obtain the type, position and angle of the target audio segment in the second multi-channel sound.
[0039] In addition, to achieve the above-mentioned objectives, the present invention also proposes a multi-channel sound recognition and positioning device, which includes: a memory, a processor, and a multi-channel sound recognition and positioning program stored in the memory and executable on the processor, wherein the multi-channel sound recognition and positioning program is configured to implement the steps of the multi-channel sound recognition and positioning method as described above.
[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a multi-channel sound recognition and positioning program is stored. When the multi-channel sound recognition and positioning program is executed by a processor, the steps of the multi-channel sound recognition and positioning method described above are implemented.
[0041] The present invention masks the GCC-PHAT features by using the Gabor features extracted through identification, that is, masking the high-dimensional features through low-dimensional features, and then using the masked GCC-PHAT features for training the network model. This can solve the problem of excessive coupling between parameters in the TCN-based SELD in the prior art, thereby effectively improving the generalization ability of the model and the recognition and positioning accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a structural schematic diagram of a multi-channel sound recognition and positioning device for a hardware operating environment involved in an embodiment of the present invention;
[0043] Figure 2 It is a flowchart of a first embodiment of a method for identifying and locating multi-channel sound according to the present invention;
[0044] Figure 3 It is a flow chart of a second embodiment of a method for identifying and locating multi-channel sound according to the present invention;
[0045] Figure 4 It is a structural block diagram of the first embodiment of the multi-channel sound recognition and positioning device of the present invention.
[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0048] Reference Figure 1 , Figure 1The present invention is a schematic diagram of the structure of a multi-channel sound recognition and positioning device in a hardware operating environment according to an embodiment of the present invention.
[0049] like Figure 1 As shown, the multi-channel sound recognition and positioning device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0050] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the multi-channel sound recognition and positioning device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.
[0051] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a multi-channel sound recognition and positioning program.
[0052] exist Figure 1 In the multi-channel sound recognition and positioning device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the multi-channel sound recognition and positioning device of the present invention can be set in the multi-channel sound recognition and positioning device, and the multi-channel sound recognition and positioning device calls the multi-channel sound recognition and positioning program stored in the memory 1005 through the processor 1001, and executes the multi-channel sound recognition and positioning method provided by the embodiment of the present invention.
[0053] The embodiment of the present invention provides a method for identifying and locating multi-channel sound, referring to Figure 2 , Figure 2The figure is a flow chart of a first embodiment of a multi-channel sound recognition and positioning method according to the present invention.
[0054] In this embodiment, the multi-channel sound recognition and positioning method includes the following steps:
[0055] Step S10: Obtaining Gabor features and GCC-PHAT features of the first multi-channel sound, wherein the Gabor features include a first label of each audio segment in the first multi-channel sound, the first label is a real type, and the GCC-PHAT features include a second label of each audio segment in the first multi-channel sound, the label information is a real position and a real angle;
[0056] Specifically, Gabor features refer to the features of the SED branch of Sound Events Detected. Gabor features can be obtained through the mel filter bank. GCC-PHAT features refer to the features of the DOA branch of Sound Events Detected. The specific steps for obtaining GCC-PHAT features are as follows:
[0057] Specifically, the IR of the sampled reverberation can be obtained by recording sampling. First, a fixed-size length sequence audio (i.e., the first multi-channel sound) is played around the microphone through a speaker, and then the IR of the length sequence audio is recorded through a microphone array. The recorded directional response is The recorded height spans θ∈[-45,45], where IR (impulse response) is a descriptive method used in acoustics to reflect the reverberation characteristics of things. After the sound source makes a sound, the sound wave begins to propagate in the room and reflects between obstacles. Each reflection is absorbed by the obstacle. In this way, when the sound source stops making sound, the sound wave will be reflected and absorbed many times in the room before finally disappearing. The sound waves are superimposed on each other to produce reverberation (the sound continuation phenomenon that still exists after the sound source in the room stops making sound is called reverberation). This period of time is called the reverberation time.
[0058] Specifically, the IR of the first multi-channel sound recorded by the microphone array can be recorded using a directional array response or a non-directional array response. The non-directional array response can be a first-order Ambisonics (FOA), and the directional array response can be a four-sided microphone array (MIC). The modeled response represents the directional response of each channel to a source incident from a direction of arrival (DOA) given by the directional response. and the pitch angle θ, wherein the direction response of the incident source is the true position of the incident source, and the direction of the incident source is the true angle of the incident source.
[0059] Among them, the first-order Ambisonics (FOA) is obtained by converting the spherical Eigenmike microphone array into a 32-channel signal through the coding filter of the anechoic measurement (assuming that the coding format is encoded in a frequency-independent manner), specifically:
[0060] H1(φ,θ,f)=1
[0061] H2(φ,θ,f)=sin(φ)*cos(θ)
[0062] H3(φ,θ,f)=sin(θ)
[0063] H4(φ,θ,f)=cos(φ)*cos(θ)
[0064] Among them, the four-sided microphone array (MIC) needs to limit the positions (φ, θ, f) of the four microphones in spherical coordinates, which are specifically defined as:
[0065] M1: (45°, 35°, 4.2cm)
[0066] M2: (-45°, -35°, 4.2cm)
[0067] M3: (135°, -35°, 4.2cm)
[0068] M4: (-135°, 35°, 4.2cm)
[0069] The analytical expansion of the directional array response is:
[0070]
[0071] In the formula, m represents the number of channels, (φ m ,θ m ) represents the directional response and elevation position of the microphone, ω = 2πf is the angular frequency, R = 0.042m is the array radius, c = 343m / s is the speed of sound, cos(γ m ) is the cosine angle between the microphone and the target, P n is the nth-degree non-normalized Legendre polynomial. h' n (2) is the derivative of the spherical Hankel function of the second kind.
[0072] Then, the GCC-PHAT feature is obtained according to the array response analytical expression. The specific calculation formula is:
[0073] x i =FFT(H m (φ,θ,ω))
[0074]
[0075]
[0076] Where τ represents the time difference from the sound source to the i'th microphone, ω k =2πk / K is the angular frequency, x i It represents the frequency spectrum of the i-th signal after short-time Fourier transform, x i' It represents the frequency spectrum of the i'th signal after short-time Fourier transform, () * represents its conjugation, R ii' (τ, t) is the generalized cross-correlation function representing the binaural channels, which is the implemented GCC-PHAT feature, cos∠x i represents the pitch angle or directional response of the ith signal, cos∠x i' represents the pitch angle or directional response of the i'th signal, and K represents the number of FFT points.
[0077] Step S20: determining a mask of the GCC-PHAT feature according to the Gabor feature, and inputting the masked GCC-PHAT feature into the first model to obtain a predicted value of each audio segment in the first multi-channel sound, wherein the predicted value includes a predicted position and a predicted angle;
[0078] Specifically, the specific steps of determining the mask of the GCC-PHAT feature according to the Gabor feature are: inputting the Gabor feature into the sigmoid layer in the resnet18 network to generate a random number; using the random number as the mask of the GCC-PHAT feature, mainly applying the generated random number to the GCC-PHAT in the form of an inner product, and the specific formula is:
[0079]
[0080] In the formula, R ii' (τ,t) represents the GCC-PHAT feature, η ii' (t, k) represents the random mask convolution kernel (random number) generated based on Gabor features.
[0081] Specifically, the masked GCC-PHAT features are input into the first model to obtain the predicted position (ie, predicted directional response) and predicted angle (ie, predicted pitch angle) of each audio segment in the first multi-channel sound. The first model is a resnet18 network.
[0082] Step S30: inputting the Gabor features into a second model to obtain a predicted category of each audio segment in the first multi-channel sound;
[0083] Specifically, the second model is a TCN network.
[0084] Step S40: Obtaining a target network model according to the first model and the second model;
[0085] Specifically, a first error is obtained according to a difference between the first label and the predicted type; a second error is obtained according to a difference between the second label and the predicted value; and the first model and the second model are updated based on the first error and the second error to obtain a target network model.
[0086] Specifically, when calculating the first error, a binary cross entropy loss function can be used to calculate the error of each audio segment. The specific calculation formula is as follows:
[0087]
[0088] Where N is the number of audio types, y i Indicates the real type, Represents the predicted category, and m represents the vector length in the true category.
[0089] Specifically, when calculating the second error, the mean square error can be used for calculation. The second error, namely, the DOA loss, can be divided into a pitch angle loss and a direction response loss. The specific calculation formula is as follows:
[0090]
[0091] In the formula, n represents the number of channels, f azi (x) represents the predicted pitch angle, y azi represents the true pitch angle, f ele (x) represents the predicted direction response, y ele represents the true directional response,
[0092] Specifically, based on the first error and the second error, a joint loss is determined; based on the joint loss, the first model and the second model are updated to obtain a target network model.
[0093] Specifically, the specific calculation formula of the joint loss is as follows:
[0094] Loss = α·Loss SED +(1-α)·Loss DOA
[0095] In the formula, Loss SED Indicates SED loss, Loss DOA represents DOA loss, α=0.02.
[0096] Specifically, the training of the first model and the second model is supervised by the joint loss to obtain a trained model (i.e., the target network model).
[0097] S50: Inputting the second multi-channel sound into the target network model to obtain the type, position and angle of the target audio segment in the second multi-channel sound.
[0098] Specifically, the second multi-channel sound can be a long audio segment, which is divided into different frames. Different frames represent different sound events. The target audio segment is one of the sound events. The position of the target audio segment can be understood as the start and end time of the sound event in the long audio. The position of the target audio segment refers to the directional response of the sound event, and the angle of the target audio segment refers to the pitch angle of the sound event.
[0099] This embodiment further extracts high-dimensional features from the acquired Gabor features, so that the network can learn the positions that require mask testing, thereby enhancing the robustness and generalization ability of the GCC-PHAT features. The actual features extracted from the GCC-PHAT features can also act on the acquired Gabor features in turn. The interaction can prevent the situation where shared weights lead to an increase in common losses, and can make full use of the features of each channel in the network, thereby improving the recognition and positioning accuracy of the network.
[0100] refer to Figure 3 , Figure 3 It is a flow chart of a second embodiment of a multi-channel sound recognition and positioning method of the present invention.
[0101] Based on the first embodiment, the method for identifying and locating multi-channel sound in this embodiment, the step of obtaining Gabor features of the first multi-channel sound includes:
[0102] Step S101: inputting the first basic features of the first multi-channel into the four convolutional layers in the Gabor filter bank to obtain static features;
[0103] Specifically, the first basic features of the first multi-channel are input into the first convolution layer in the four convolution layers to obtain the second basic features; the second basic features are input into the global mean pooling layer in the four convolution layers to obtain the third basic features; the third basic features are input into the Gaussian low-pass filter layer in the four convolution layers to obtain the fourth basic features; the fourth basic features are input into the normalization layer in the four convolution layers to obtain the static features.
[0104] Specifically, when the first basic feature is input into the first convolutional layer, the learned feature is constrained to a certain range to obtain the second basic feature, and then the second basic feature is input into the global mean pooling layer to obtain the Hilbert envelope (i.e., the third basic feature), and the third basic feature is input into the Gaussian low-pass filter layer to perform noise reduction on the third basic feature to obtain the fourth basic feature, and the fourth basic feature is input into the normalization layer to obtain the final static feature using multi-rate multi-channel normalization.
[0105] Step S102: inputting the static features into the differential operation layer in the Gabor filter bank to obtain dynamic features;
[0106] Specifically, the static features are first input into the differential operation layer for a first-order differential operation, and then a second-order differential operation. The dynamic features are obtained by jointly obtaining the first-order differential of the static features and the second-order differential of the static features.
[0107] Step S103: determining Gabor features of the first channel sound based on the static features and the dynamic features.
[0108] Specifically, by combining the static features and the dynamic features, Gabor features can be obtained.
[0109] This embodiment abandons the original method of extracting static features through the mel filter group and adopts a constrained four-layer convolution layer to extract static features, which can achieve constrained positioning and recognition. It can not only effectively reduce the reduction in positioning accuracy caused by shared weights, but also retain the dependence of the positioning features on the correlation of the recognition features, further improving the generalization ability of the model and the recognition and positioning accuracy of the model.
[0110] Reference Figure 4 , Figure 4 It is a structural block diagram of the first embodiment of the multi-channel sound recognition and positioning device of the present invention.
[0111] like Figure 4 As shown, the multi-channel sound recognition and positioning device proposed in the embodiment of the present invention includes:
[0112] An acquisition module 100 is used to acquire Gabor features and GCC-PHAT features of a first multi-channel sound;
[0113] The acquisition module 100 is further used to determine a mask of the GCC-PHAT feature according to the Gabor feature, and input the masked GCC-PHAT feature into the first model to obtain a prediction value of each audio segment in the first multi-channel sound;
[0114] The acquisition module 100 is further used to input the Gabor feature into a second model to obtain a predicted type of each audio segment in the first multi-channel sound;
[0115] The acquisition module 100 is further used to obtain a target network model according to the first model and the second model;
[0116] The identification and positioning module 200 is used to input the second multi-channel sound into the target network model to obtain the type, position and angle of the target audio segment in the second multi-channel sound.
[0117] In one embodiment, the acquisition module 100 is further used for:
[0118] Inputting the first basic features of the first multi-channel into four convolutional layers in the Gabor filter bank to obtain static features;
[0119] Inputting the static features into the differential operation layer in the Gabor filter bank to obtain dynamic features;
[0120] Based on the static feature and the dynamic feature, a Gabor feature of the first channel sound is determined.
[0121] In one embodiment, the acquisition module 100 is further used for:
[0122] Inputting the first basic features of the first multi-channels into the first convolutional layer of the four convolutional layers to obtain second basic features;
[0123] Inputting the second basic feature into the global mean pooling layer in the four convolutional layers to obtain a third basic feature;
[0124] Inputting the third basic feature into the Gaussian low-pass filter layer in the four convolutional layers to obtain a fourth basic feature;
[0125] The fourth basic feature is input into a normalization layer in the four convolutional layers to obtain a static feature.
[0126] In one embodiment, the acquisition module 100 is further used for:
[0127] Obtain a second label for each audio segment in the first multi-channel sound;
[0128] After determining the array response of each audio segment based on the second label, a GCC-PHAT feature of the first multi-channel sound is determined according to the array response.
[0129] In one embodiment, the acquisition module 100 is further used for:
[0130] Input the Gabor features into the sigmoid layer in the resnet18 network to generate random numbers;
[0131] The random number is used as a mask for the GCC-PHAT feature.
[0132] In one embodiment, the acquisition module 100 is further used for:
[0133] Obtaining a first error according to a difference between the first label and the predicted category;
[0134] Obtaining a second error according to a difference between the second label and the predicted value;
[0135] The first model and the second model are updated based on the first error and the second error to obtain a target network model.
[0136] In one embodiment, the acquisition module 100 is further used for:
[0137] determining a joint loss based on the first error and the second error;
[0138] The first model and the second model are updated based on the joint loss to obtain a target network model.
[0139] In addition, an embodiment of the present invention also proposes a multi-channel sound recognition and positioning device, which includes a memory, a processor, and a multi-channel sound recognition and positioning program stored in the memory and executable on the processor, wherein the multi-channel sound recognition and positioning program is configured to implement the steps of the multi-channel sound recognition and positioning method as described above.
[0140] In addition, an embodiment of the present invention further proposes a storage medium, on which a program for avoiding traffic jams is stored. When the multi-channel sound recognition and positioning program is executed by a processor, the steps of the multi-channel sound recognition and positioning method described above are implemented.
[0141] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.
[0142] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0143] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the method for avoiding traffic congestion provided in any embodiment of the present invention, and will not be repeated here.
[0144] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0145] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0147] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for identifying and locating multi-channel sound, characterized in that: The multi-channel sound recognition and positioning method comprises: Obtaining Gabor features and GCC-PHAT features of the first multi-channel sound, wherein the Gabor features include a first label of each audio segment in the first multi-channel sound, the first label is a real type, and the GCC-PHAT features include a second label of each audio segment in the first multi-channel sound, the second label is a real position and a real angle; Inputting the Gabor features into a first model to obtain a predicted category of each audio segment in the first multi-channel sound; Determine a mask of the GCC-PHAT feature according to the Gabor feature, and input the masked GCC-PHAT feature into a second model to obtain a predicted value of each audio segment in the first multi-channel sound, wherein the predicted value includes a predicted position and a predicted angle; Obtain a target network model according to the first model and the second model; The second multi-channel sound is input into the target network model to obtain the type, position and angle of the target audio segment in the second multi-channel sound.
2. The multi-channel sound recognition and positioning method according to claim 1, characterized in that: The obtaining of Gabor features of the first multi-channel sound includes: Inputting the first basic features of the first multi-channel into four convolutional layers in the Gabor filter bank to obtain static features; Inputting the static features into the differential operation layer in the Gabor filter bank to obtain dynamic features; Based on the static features and the dynamic features, Gabor features of the first multi-channel sound are determined.
3. The multi-channel sound recognition and positioning method according to claim 2, characterized in that: The step of inputting the first basic features of the first multi-channel into the four convolutional layers in the Gabor filter bank to obtain static features includes: Inputting the first basic features of the first multi-channels into the first convolutional layer of the four convolutional layers to obtain second basic features; Inputting the second basic feature into the global mean pooling layer in the four convolutional layers to obtain a third basic feature; Inputting the third basic feature into the Gaussian low-pass filter layer in the four convolutional layers to obtain a fourth basic feature; The fourth basic feature is input into a normalization layer in the four convolutional layers to obtain a static feature.
4. The multi-channel sound recognition and positioning method according to claim 1, characterized in that: The obtaining of the GCC-PHAT feature of the first multi-channel sound comprises: Obtain a second label for each audio segment in the first multi-channel sound; After determining the array response of each audio segment based on the second label, a GCC-PHAT feature of the first multi-channel sound is determined according to the array response.
5. The multi-channel sound recognition and positioning method according to any one of claims 1 to 4, characterized in that: The step of determining the mask of the GCC-PHAT feature according to the Gabor feature comprises: Input the Gabor features into the sigmoid layer in the resnet18 network to generate random numbers; The random number is used as a mask for the GCC-PHAT feature.
6. The multi-channel sound recognition and positioning method according to any one of claims 1 to 4, characterized in that: Obtaining a target network model according to the first model and the second model includes: Obtaining a first error according to a difference between the first label and the predicted category; Obtaining a second error according to a difference between the second label and the predicted value; The first model and the second model are updated based on the first error and the second error to obtain a target network model.
7. The multi-channel sound recognition and positioning method according to claim 6, characterized in that: The updating of the first model and the second model based on the first error and the second error to obtain a target network model includes: determining a joint loss based on the first error and the second error; The first model and the second model are updated based on the joint loss to obtain a target network model.
8. A multi-channel sound recognition and positioning device, characterized in that: The multi-channel sound recognition and positioning device comprises: An acquisition module, used for acquiring Gabor features and GCC-PHAT features of the first multi-channel sound; The acquisition module is further used to input the Gabor feature into the first model to obtain the predicted type of each audio segment in the first multi-channel sound; The acquisition module is further used to determine a mask of the GCC-PHAT feature according to the Gabor feature, and input the masked GCC-PHAT feature into a second model to obtain a predicted value of each audio segment in the first multi-channel sound; The acquisition module is further used to obtain a target network model according to the first model and the second model; The identification and positioning module is used to input the second multi-channel sound into the target network model to obtain the type, position and angle of the target audio segment in the second multi-channel sound.
9. A multi-channel sound recognition and positioning device, characterized in that: The device includes: a memory, a processor, and a multi-channel sound recognition and positioning program stored in the memory and executable on the processor, wherein the multi-channel sound recognition and positioning program is configured to implement the steps of the multi-channel sound recognition and positioning method as described in any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores a multi-channel sound recognition and positioning program, which, when executed by a processor, implements the steps of the multi-channel sound recognition and positioning method according to any one of claims 1 to 7.
