Audio signal processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an audio signal processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a plurality of first environment audio signals, wherein each first environment audio signal comprises sound emitted by at least one sound source; determining the direction of each sound source based on the plurality of first environment audio signals; the multiple first environment audio signals are recognized based on the directions of the sound sources, sound source audio signals corresponding to all the sound sources are obtained, and all the sound source audio signals comprise sounds emitted by the sound sources corresponding to the directions of the sound sources; identifying the type of a sound source corresponding to each sound source audio signal; and displaying prompt information of the sound source in a display interface according to the direction of the sound source and the category of the sound source. According to the method, the direction and the category of the sound source are visually displayed to the user through the display interface, so that the user can know the sound condition in the surrounding environment under the condition that the user cannot hear the sound, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to a method, device, electronic device and storage medium for processing an audio signal. Background Art

[0002] With the continuous development of artificial intelligence and audio processing technology, the application scope of audio processing technology is also expanding. At present, audio processing technology has been widely used in speech recognition, speech synthesis, audio encoding and decoding, audio enhancement, audio analysis and other fields. Among them, speech recognition technology has been used in intelligent voice assistants, intelligent customer service, smart homes and other aspects. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a method, device, electronic device and storage medium for processing an audio signal.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for processing an audio signal is provided, the method comprising:

[0005] Acquire multiple first ambient audio signals, each of which includes a sound emitted by at least one sound source, and the multiple first ambient audio signals are collected by different microphones;

[0006] Based on the plurality of first ambient audio signals, determining a direction of each of the sound sources;

[0007] Identify the plurality of first ambient audio signals based on the direction of the sound source to obtain a sound source audio signal corresponding to each of the sound sources, wherein each sound source audio signal includes a sound emitted by a sound source corresponding to the direction of the sound source;

[0008] Identifying the category of the sound source corresponding to each of the sound source audio signals;

[0009] Prompt information of the sound source is displayed in a display interface according to the direction of the sound source and the category of the sound source, and the prompt information is used to represent the direction of each sound source and the category to which the sound emitted by the sound source belongs.

[0010] In some embodiments, determining the direction of each of the sound sources based on the plurality of first ambient audio signals comprises:

[0011] Acquire signal characteristics of the multiple first ambient audio signals, where the signal characteristics are used to characterize multiple time-frequency points and phase information of each of the first ambient audio signals, and multiple time-frequency points in the multiple first ambient audio signals are the same;

[0012] Predicting the direction of the sound source based on the signal characteristics to determine direction information, the direction information including the probability of each of the preset directions at the multiple time-frequency points, the probability of the preset direction at the time-frequency point indicating the possibility that the preset direction is the direction of the time-frequency point;

[0013] Based on the direction information, the direction of each of the sound sources is determined.

[0014] In some embodiments, obtaining signal characteristics of the plurality of first ambient audio signals includes:

[0015] Determine the multiple time-frequency points based on the number of time frames and the number of frequency points of any one of the first ambient audio signals;

[0016] Selecting any one of the plurality of first ambient audio signals as the second ambient audio signal;

[0017] The phase information of each of the first ambient audio signals at each of the time-frequency points is determined according to the relationship between each of the first ambient audio signals and the second ambient audio signals at each of the time-frequency points.

[0018] In some embodiments, determining the phase information of each of the first ambient audio signals at each time-frequency point according to the relationship between each of the first ambient audio signals and the second ambient audio signals at each of the time-frequency points includes:

[0019] For each of the first ambient audio signals, determining a ratio of the first ambient audio signal to the second ambient audio signal at each of the time-frequency points, wherein the ratio of the time-frequency points is used to characterize a phase of the first ambient audio signal at the time-frequency point;

[0020] A phase corresponding to the ratio of each of the time-frequency points is determined, and the phases of the multiple time-frequency points are used as the phase information.

[0021] In some embodiments, the predicting the direction of the sound source based on the signal feature and determining the direction information includes:

[0022] The sound source localization model is called to predict the direction of the sound source for the signal feature to obtain the direction information. The sound source localization model is used to predict the direction of any sound source.

[0023] In some embodiments, determining the direction of each of the sound sources based on the direction information includes:

[0024] Determine a third ambient audio signal based on the multiple first ambient audio signals, where the third ambient audio signal is used to represent the multiple first ambient audio signals;

[0025] For each of the preset directions, based on the probability of the preset direction at the multiple time-frequency points and the third ambient audio signal, determining the energy corresponding to the preset direction, wherein the energy corresponding to the preset direction is used to characterize the possibility that the preset direction is the direction of any sound source;

[0026] Based on the energies corresponding to the multiple preset directions, the direction of each of the sound sources is determined, and the energy corresponding to the direction of each of the sound sources is greater than the energies corresponding to other preset directions.

[0027] In some embodiments, the identifying the plurality of first ambient audio signals based on the direction of the sound source to obtain a sound source audio signal corresponding to each of the sound sources includes:

[0028] Based on the multiple first ambient audio signals, determine a fourth ambient audio signal, where the fourth ambient audio signal is used to characterize the multiple first ambient audio signals;

[0029] Based on the direction information and the direction of the sound source, a sound source audio signal corresponding to each of the sound sources is identified from the fourth ambient audio signal.

[0030] In some embodiments, the identifying, from the fourth ambient audio signal, a sound source audio signal corresponding to each of the sound sources based on the direction information and the direction of the sound source comprises:

[0031] For each sound source, multiply the probability of the direction of the sound source at each time-frequency point by the amplitude of each time-frequency point in the fourth ambient audio signal to obtain the amplitude of the sound emitted by the sound source;

[0032] Using the phase of each of the time-frequency points in the fourth ambient audio signal as the phase of the sound emitted by the sound source;

[0033] Based on the amplitude and the phase, a sound source audio signal corresponding to the sound source is determined.

[0034] In some embodiments, the identifying the category of the sound source corresponding to each of the sound source audio signals comprises:

[0035] Calling a category recognition model to recognize each of the sound source audio signals and obtain the category of the sound source corresponding to each of the sound source audio signals, wherein the category recognition model is trained based on a training data set, the training data set includes sample sound source audio signals of multiple categories and a sample category corresponding to each of the sample sound source audio signals, the sample sound source audio signals of the category are sound source audio signals belonging to the category cut out from the sample audio signals, and the difference between the numbers of sample sound source audio signals of different categories is less than a preset value.

[0036] In some embodiments, the prompt information is a sound source category icon, and the displaying of the prompt information of the sound source in the display interface includes:

[0037] A sound source category icon corresponding to the sound source is displayed in the display interface, wherein different categories of sound sources correspond to different sound source category icons, and the position of the sound source category icon in the display interface indicates the direction of the sound source.

[0038] In some embodiments, the display interface includes a coordinate graph, and the prompt information includes a sound source category icon; and displaying the prompt information of the sound source in the display interface includes:

[0039] Determining a sound source category icon corresponding to the category of the search sound source according to the category of the sound source;

[0040] Determine the target coordinate position according to the direction of the sound source;

[0041] The sound source category icon is displayed at the target coordinate position on the coordinate graph.

[0042] According to a second aspect of an embodiment of the present disclosure, there is provided an audio signal processing device, the device comprising:

[0043] a signal acquisition module, configured to acquire a plurality of first ambient audio signals, each of which includes a sound emitted by at least one sound source, and the plurality of first ambient audio signals are collected by different microphones;

[0044] a direction determination module, configured to determine the direction of each of the sound sources based on the plurality of first ambient audio signals;

[0045] a signal recognition module configured to recognize the plurality of first ambient audio signals based on the direction of the sound source, and obtain a sound source audio signal corresponding to each of the sound sources, wherein each sound source audio signal includes a sound emitted by a sound source corresponding to the direction of the sound source;

[0046] A category identification module, configured to identify the category of the sound source corresponding to each of the sound source audio signals;

[0047] The display module is configured to display prompt information of the sound source in a display interface according to the direction of the sound source and the category of the sound source, wherein the prompt information is used to characterize the direction of each sound source and the category to which the sound emitted by the sound source belongs.

[0048] In some embodiments, the direction determination module is configured to:

[0049] Acquire signal characteristics of the multiple first ambient audio signals, where the signal characteristics are used to characterize multiple time-frequency points and phase information of each of the first ambient audio signals, and multiple time-frequency points in the multiple first ambient audio signals are the same;

[0050] Predicting the direction of the sound source based on the signal characteristics to determine direction information, the direction information including the probability of each of the preset directions at the multiple time-frequency points, the probability of the preset direction at the time-frequency point indicating the possibility that the preset direction is the direction of the time-frequency point;

[0051] Based on the direction information, the direction of each of the sound sources is determined.

[0052] In some embodiments, the direction determination module is configured to:

[0053] Determine the multiple time-frequency points based on the number of time frames and the number of frequency points of any one of the first ambient audio signals;

[0054] Selecting any one of the plurality of first ambient audio signals as the second ambient audio signal;

[0055] The phase information of each of the first ambient audio signals at each of the time-frequency points is determined according to the relationship between each of the first ambient audio signals and the second ambient audio signals at each of the time-frequency points.

[0056] In some embodiments, the direction determination module is configured to:

[0057] For each of the first ambient audio signals, determining a ratio of the first ambient audio signal to the second ambient audio signal at each of the time-frequency points, wherein the ratio of the time-frequency points is used to characterize a phase of the first ambient audio signal at the time-frequency point;

[0058] A phase corresponding to the ratio of each of the time-frequency points is determined, and the phases of the multiple time-frequency points are used as the phase information.

[0059] In some embodiments, the direction determination module is configured to:

[0060] The sound source localization model is called to predict the direction of the sound source for the signal feature to obtain the direction information. The sound source localization model is used to predict the direction of any sound source.

[0061] In some embodiments, the direction determination module is configured to:

[0062] Determine a third ambient audio signal based on the multiple first ambient audio signals, where the third ambient audio signal is used to represent the multiple first ambient audio signals;

[0063] For each of the preset directions, based on the probability of the preset direction at the multiple time-frequency points and the third ambient audio signal, determining the energy corresponding to the preset direction, wherein the energy corresponding to the preset direction is used to characterize the possibility that the preset direction is the direction of any sound source;

[0064] Based on the energies corresponding to the multiple preset directions, the direction of each of the sound sources is determined, and the energy corresponding to the direction of each of the sound sources is greater than the energies corresponding to other preset directions.

[0065] In some embodiments, the direction determination module is configured to:

[0066] Based on the multiple first ambient audio signals, determine a fourth ambient audio signal, where the fourth ambient audio signal is used to characterize the multiple first ambient audio signals;

[0067] Based on the direction information and the direction of the sound source, a sound source audio signal corresponding to each of the sound sources is identified from the fourth ambient audio signal.

[0068] In some embodiments, the direction determination module is configured to:

[0069] For each sound source, multiply the probability of the direction of the sound source at each time-frequency point by the amplitude of each time-frequency point in the fourth ambient audio signal to obtain the amplitude of the sound emitted by the sound source;

[0070] Using the phase of each of the time-frequency points in the fourth ambient audio signal as the phase of the sound emitted by the sound source;

[0071] Based on the amplitude and the phase, a sound source audio signal corresponding to the sound source is determined.

[0072] In some embodiments, the category identification module is configured to:

[0073] Calling a category recognition model to recognize each of the sound source audio signals and obtain the category of the sound source corresponding to each of the sound source audio signals, wherein the category recognition model is trained based on a training data set, the training data set includes sample sound source audio signals of multiple categories and a sample category corresponding to each of the sample sound source audio signals, the sample sound source audio signals of the category are sound source audio signals belonging to the category cut out from the sample audio signals, and the difference between the numbers of sample sound source audio signals of different categories is less than a preset value.

[0074] In some embodiments, the prompt information is a sound source category icon, and the display module is configured to:

[0075] A sound source category icon corresponding to the sound source is displayed in the display interface, wherein different categories of sound sources correspond to different sound source category icons, and the position of the sound source category icon in the display interface indicates the direction of the sound source.

[0076] In some embodiments, the display interface includes a coordinate graph, the prompt information includes a sound source category icon; and the display module is configured to:

[0077] Determining a sound source category icon corresponding to the category of the search sound source according to the category of the sound source;

[0078] Determine the target coordinate position according to the direction of the sound source;

[0079] The sound source category icon is displayed at the target coordinate position on the coordinate graph.

[0080] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0081] processor;

[0082] a memory for storing processor-executable instructions;

[0083] The processor is configured to execute the method as described in the first aspect of the embodiment of the present disclosure.

[0084] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect of the embodiment of the present disclosure.

[0085] The above method of the present disclosure has the following beneficial effects:

[0086] The method provided by the embodiment of the present disclosure can determine the direction of the sound source in the ambient audio signal based on the collected ambient audio signal, and identify the sound source audio signal corresponding to the sound source from the ambient audio signal, and then identify the category of the sound source, and display prompt information of the sound source in the display interface. The prompt information is used to characterize the direction and category of the sound source. In this way, the direction and category of the sound source in the ambient audio signal are determined, and the direction and category of the sound source are intuitively displayed to the user through the display interface, so that the user can understand the sound situation in the surrounding environment even if he cannot hear the sound, thereby improving the user experience.

[0087] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0089] Figure 1 is a flowchart of a method for processing an audio signal according to an exemplary embodiment;

[0090] Figure 2 is a flowchart of another method for processing an audio signal according to an exemplary embodiment;

[0091] Figure 3 is a schematic diagram showing a model structure of a sound source localization model according to an exemplary embodiment;

[0092] Figure 4 is a flowchart of another method for processing an audio signal according to an exemplary embodiment;

[0093] Figure 5 is a schematic diagram showing a model structure of a category recognition model according to an exemplary embodiment;

[0094] Figure 6 is a schematic diagram of a radar chart according to an exemplary embodiment;

[0095] Figure 7 is a schematic diagram showing a process of processing an audio signal according to an exemplary embodiment;

[0096] Figure 8 is a block diagram of a device for processing an audio signal according to an exemplary embodiment;

[0097] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0098] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0099] The audio signal processing method provided in the embodiment of the present disclosure is executed by an electronic device, which may be a mobile phone, a tablet computer, a laptop computer, a wearable device, a handheld device, a smart home device, a vehicle-mounted terminal or the like, and the electronic device has a sound collection function and a display function.

[0100] Figure 1 is a flowchart of a method for processing an audio signal according to an exemplary embodiment, which is executed by an electronic device, see Figure 1 , the method comprises the following steps:

[0101] Step S101 : acquiring a plurality of first ambient audio signals, each of which includes a sound emitted by at least one sound source, and the plurality of first ambient audio signals are collected by different microphones.

[0102] Among them, the electronic device includes multiple microphones, or the electronic device is connected to multiple microphones, the multiple microphones are distributed in different positions in the electronic device, and the multiple microphones together form a microphone array. Each microphone can collect sounds in the surrounding environment to obtain a first ambient audio signal corresponding to the microphone.

[0103] Since the microphone collects different sounds in the surrounding environment at the same time, the first ambient audio signal is a mixed audio signal, which includes sounds emitted by at least one sound source, and the sounds emitted by the at least one sound source are sounds at different locations in the surrounding environment. The first ambient audio signal can be an audio signal in mp4 format, wav format or other formats.

[0104] Step S102: determining the direction of each sound source based on a plurality of first ambient audio signals.

[0105] The direction of the sound source is a direction relative to a preset object, and the preset object is the current electronic device, or a user holding or wearing the electronic device.

[0106] Since the multiple microphones are distributed at different positions in the electronic device, the multiple first ambient audio signals collected by the multiple microphones contain spatial information of the sound emitted by the sound source. Therefore, the direction of the sound source can be determined based on the multiple first ambient audio signals.

[0107] Step S103 : identifying a plurality of first ambient audio signals based on the direction of the sound source, and obtaining a sound source audio signal corresponding to each sound source.

[0108] Each sound source audio signal includes the sound emitted by the sound source corresponding to the direction of the sound source, that is, the sound source audio signal includes the sound emitted by the sound source corresponding to the sound source audio signal. Each first ambient audio signal includes the sound emitted by the sound source, so the sound source audio signal corresponding to the sound source can be identified from the first ambient audio signal.

[0109] Step S104: identifying the type of the sound source corresponding to each sound source audio signal.

[0110] Among them, the categories include human voices, car sounds, animal calls, water sounds, horn sounds, TV sounds and many other sound categories.

[0111] Step S105: displaying prompt information of the sound source in the display interface according to the direction of the sound source and the type of the sound source.

[0112] The display interface is used to display prompt information of the sound source, and the prompt information is used to represent the direction of the sound source and the category of the sound emitted by the sound source. The prompt information can be in the form of text, icon or other forms.

[0113] The method provided by the embodiment of the present disclosure can determine the direction of the sound source in the ambient audio signal based on the collected ambient audio signal, and identify the sound source audio signal corresponding to the sound source from the ambient audio signal, and then identify the category of the sound source, and display prompt information of the sound source in the display interface. The prompt information is used to characterize the direction and category of the sound source. In this way, the direction and category of the sound source in the ambient audio signal are determined, and the direction and category of the sound source are intuitively displayed to the user through the display interface, so that the user can understand the sound situation in the surrounding environment even if he cannot hear the sound, thereby improving the user experience.

[0114] The audio signal processing method provided in the embodiments of the present disclosure involves the process of collecting environmental audio signals, the process of determining the direction of the sound source, the process of identifying the sound emitted by the sound source, the process of identifying the category of the sound, and the process of displaying prompt information. These processes are respectively explained through the following embodiments.

[0115] The step of acquiring a plurality of first ambient audio signals in the above step S101 can be implemented by the following implementations:

[0116] The electronic device includes a plurality of microphones, or the electronic device is connected to a plurality of microphones, and the plurality of microphones form a microphone array. The subsequent audio processing process has no restrictions on the array of the microphone array, and the flexibility is high. The microphone array can be any array, for example, the microphone array is a linear array, a circular array or an array in other forms. Each microphone collects the sound in the surrounding environment to obtain a first ambient audio signal collected by the microphone. For example, the electronic device includes M microphones. Assuming that there are N sound sources in the surrounding environment, M is an integer greater than 1, and N is a positive integer, then the first ambient audio signal collected by the mth microphone among the M microphones can be expressed as the following formula (1):

[0117]

[0118] Among them, z m (t) represents the first ambient audio signal collected by the mth microphone, i represents the i-th sound source among N sound sources, si (t) represents the sound emitted by the i-th sound source, represents the room impulse response between the i-th sound source and the m-th microphone, * represents convolution, and t represents time.

[0119] The first ambient audio signal directly collected by the microphone is a time domain signal. When performing subsequent audio signal processing, a time-frequency domain signal is required. In some embodiments, a short-time Fourier transform is used to convert the first ambient audio signal in the time domain into the first ambient audio signal in the time-frequency domain. The first ambient audio signal in the time-frequency domain can be expressed as the following formula (2):

[0120]

[0121] Among them, z m (l,k) represents z m (t) corresponds to the first ambient audio signal in the time-frequency domain, s i (l,k) represents s i (t) corresponds to the time-frequency domain sound, express The corresponding room impulse response in the time-frequency domain, l represents the lth time frame, and k represents the kth frequency point.

[0122] The step of determining the direction of each sound source based on the plurality of first ambient audio signals in step S102 can be performed as follows: Figure 2 The illustrated implementation achieves:

[0123] Figure 2 is a flowchart of another method for processing an audio signal according to an exemplary embodiment, which is executed by an electronic device, see Figure 2 , the method comprises the following steps:

[0124] Step S201: Acquire signal characteristics of a plurality of first ambient audio signals.

[0125] The signal feature is used to characterize multiple time-frequency points and phase information of each first ambient audio signal, and multiple time-frequency points in multiple first ambient audio signals are the same.

[0126] In some embodiments, based on the number of time frames and the number of frequency points of any first ambient audio signal, a plurality of time-frequency points are determined. That is, when the first ambient audio signal is a time-frequency domain signal, the number of time frames and the number of frequency points can be directly obtained from the first ambient audio signal. For example, the first ambient audio signal z shown in the above formula (2) is m(l,k), l represents the lth time frame, k represents the kth frequency point, and (l,k) represents a time-frequency point. Then, for the first ambient audio signal, multiple time-frequency points in the first ambient audio signal can be determined. Moreover, since multiple microphones collect the same sound, multiple time-frequency points in multiple first ambient audio signals are also the same.

[0127] Optionally, the number of time frames in the first ambient audio signal is L, the number of frequency points is K, and in any time-frequency point (l, k), l is less than L, and k is less than K, where L and K are integers greater than 1.

[0128] In some embodiments, any first ambient audio signal is selected from a plurality of first ambient audio signals as a second ambient audio signal, and the second ambient audio signal is a reference audio signal; and the phase information of each first ambient audio signal at each time-frequency point is determined based on the relationship between each first ambient audio signal and the second ambient audio signal at each time-frequency point. For example, the phase information is estimated using an instantaneous relative transfer function. It should be noted that, for the selected first ambient audio signal, the phase information of the selected first ambient audio signal at each time-frequency point is no longer determined.

[0129] Optionally, according to the relationship between each first ambient audio signal and the second ambient audio signal at each time-frequency point, the phase information of each first ambient audio signal at each time-frequency point is determined, including: for each first ambient audio signal, determining the ratio of the first ambient audio signal to the second ambient audio signal at each time-frequency point, the ratio of the time-frequency point is used to characterize the phase of the first ambient audio signal at the time-frequency point; determining the phase corresponding to the ratio of each time-frequency point, and using the phases of multiple time-frequency points as phase information. For example, the following formula (3) is used to represent the phase of the first ambient audio signal collected by the mth microphone at the time-frequency point (l, k):

[0130]

[0131] Among them, r(l,k,m) represents the phase of the first ambient audio signal collected by the mth microphone at the time-frequency point (l,k), z m (l, k) represents the first ambient audio signal in the time-frequency domain collected by the mth microphone, z ref (l, k) represents the second ambient audio signal, Indicates an angle.

[0132] In the above formula (3), the sine value and the cosine value are calculated, and both the sine value and the cosine value are used as the values ​​representing the phase of the time-frequency point. That is, the first ambient audio signal has two values ​​representing the phase at any time-frequency point.

[0133] In some embodiments, an L×K matrix R is determined based on the number of time frames L and the number of frequency points K of the first ambient audio signal, wherein each entry (l, k) in the matrix R includes M channels, each channel corresponds to a microphone, wherein each channel is used to represent the phase information of the channel corresponding to the current time-frequency point (l, k). Optionally, when the first ambient audio signal has two values ​​representing the phase at any time-frequency point, each entry in the matrix R contains 2(M-1) values ​​representing the phase, and when the matrix R is used as a signal feature, the signal feature is a feature of (L, K, 2(M-1)) dimensions.

[0134] In other embodiments, the signal sub-feature of each first ambient audio signal can be determined according to the multiple time-frequency points of each first ambient audio signal and the phase information of each first ambient audio signal, and each signal sub-feature is a feature of (L, K, 2) dimensions. For multiple first ambient audio signals that do not include the selected first ambient audio signal, the multiple signal sub-features of the multiple first ambient audio signals are combined to obtain a feature of (L, K, 2(M-1)) dimensions.

[0135] Step S202: predict the direction of the sound source based on the signal characteristics to determine the direction information.

[0136] The direction information includes the probability of each preset direction in multiple time-frequency points, and the probability of the preset direction in the time-frequency point indicates the possibility that the preset direction is the direction of the time-frequency point. The preset direction is a plurality of preset angles, each angle representing a direction, for example, the preset direction θ may be {0°, 15°, 30°, ..., 345°}, and for another example, the preset direction θ may be {0°, 10°, 20°, ..., 350°}.

[0137] For example, the following formula (4) is used to express the probability of the preset direction θ at the time-frequency point (l, k):

[0138] p l,k (θ) = p(y l,k =θ|R) (4)

[0139] Among them, p l,k (θ) represents the probability of the preset direction θ at the time-frequency point (l, k), R represents the signal feature, y l,k Indicates the direction of the time-frequency point (l,k).

[0140] In some embodiments, a sound source localization model is called to predict the direction of the sound source based on the signal features to obtain direction information, wherein the sound source localization model is used to predict the direction of any sound source.

[0141] In some embodiments, the sound source localization model includes an encoder and a decoder, and the encoder and the decoder are connected to each other through skip connections. The encoder is composed of multiple convolutional layers and pooling layers, which are used to extract the coding features of the input feature map (using the signal features as a feature map). Each convolutional layer includes a convolution (3*3Conv) operation, a batch normalization (BN, Batch Normalization) operation and an activation function (ELU, Exponential Linear Unit). The convolutional layer is used to extract the high-level features of the feature map, and each pooling layer is used to reduce the resolution of the feature map to reduce the amount of calculation and memory consumption. The decoder is composed of multiple deconvolution layers, multiple convolutional layers and cross-layer connections, which are used to convert the coded features extracted by the toilet into a segmentation feature map with the same resolution as the input feature map. Each deconvolution layer includes a deconvolution (2*2De-Conv) operation, a batch normalization operation and an activation function for upsampling and reconstructing the feature map. The cross-layer connection is used to connect the feature maps between the encoder and the encoder to retain more low-level feature information. The decoder finally includes a 1*1 convolution layer for obtaining the model output. In the disclosed embodiment, the sound source localization model outputs the probability of each of the multiple preset directions at multiple time-frequency points.

[0142] For example, see Figure 3 The schematic diagram of the model structure of the sound source localization model shown in Figure 3In the method, the signal feature of (L, K, 2(M-1)) dimension is input into the sound source localization model, and then the signal feature passes through two convolution layers and one pooling layer to obtain the first coding feature, the first coding feature passes through two convolution layers and one pooling layer to obtain the second coding feature, the second coding feature passes through two convolution layers and one pooling layer to obtain the third coding feature, the third coding feature passes through two convolution layers and one pooling layer to obtain the fourth coding feature. The fourth encoding feature passes through two convolutional layers and a deconvolution layer to obtain the first decoding feature. Based on cross-layer connection, the first decoding feature and the fourth encoding feature are combined to obtain the first combined feature. The first combined feature passes through two convolutional layers and a deconvolution layer to obtain the second decoding feature. Based on cross-layer connection, the second decoding feature and the third encoding feature are combined to obtain the second combined feature. The second combined feature passes through two convolutional layers and a deconvolution layer to obtain the third decoding feature. Based on cross-layer connection, the third decoding feature and the second encoding feature are combined to obtain the third combined feature. The third combined feature passes through two convolutional layers and a deconvolution layer to obtain the fourth decoding feature. Based on cross-layer connection, the third decoding feature and the first encoding feature are combined to obtain the fourth combined feature. The fourth combined feature passes through two convolutional layers and a 1*1 convolution layer to obtain the model output.

[0143] It should be noted that Figure 3 The process of performing four encodings and four decodings is merely taken as an example for explanation. In another embodiment, the sound source localization model may be encoded and decoded X times to obtain a model output.

[0144] For the training process of the sound source localization model, optionally, sample signal features of multiple sample environmental audio signals and multiple sample probabilities are obtained, where the sample probability is the probability of a preset direction at a time-frequency point, and the sample signal features are input into the sound source localization model. The sound source localization model outputs multiple prediction probabilities, and a cross-entropy cost function is used based on the multiple prediction probabilities and the multiple sample probabilities to determine a loss value. The model parameters of the sound source localization model are adjusted based on the loss value, and the sound source localization model is iteratively trained multiple times using the above method to obtain a trained sound source localization model.

[0145] Optionally, the sound source localization model is trained using the ADAM optimizer (a model optimization algorithm).

[0146] Step S203: determining the direction of each sound source based on the direction information.

[0147] According to the w-disjoint orthogonality assumption, each time-frequency point has a certain DOA (direction of arrival), that is, each time-frequency point has a certain direction. For multiple time-frequency points, the directions of different time-frequency points may be the same or different.

[0148] In some embodiments, based on multiple first ambient audio signals, a third ambient audio signal is determined, and for each preset direction, the energy corresponding to the preset direction is determined based on the probability of the preset direction at multiple time-frequency points and the third ambient audio signal; based on the energies corresponding to multiple preset directions, the direction of each sound source is determined, and the energy corresponding to the direction of each sound source is greater than the energies corresponding to other preset directions.

[0149] Among them, the third ambient audio signal is used to characterize multiple first ambient audio signals. The third ambient audio signal can be any audio signal among the multiple first ambient audio signals, or it can be an average audio signal of the multiple first ambient audio signals. The third ambient audio signal can be the same as or different from the above-mentioned second ambient audio signal.

[0150] The energy corresponding to the preset direction is used to characterize the possibility that the preset direction is the direction of any sound source. The larger the energy, the greater the possibility that there is a sound source in the preset direction corresponding to the energy. The smaller the energy, the smaller the possibility that there is a sound source in the preset direction corresponding to the energy. Therefore, the preset direction with the largest corresponding energy is selected as the direction of the sound source.

[0151] For example, referring to the following formulas (5) and (6), N preset directions with the largest energy are selected:

[0152]

[0153] E(θ1)≥E(θ2)≥E(θ3)≥…≥E(θ N ) (6)

[0154] Among them, E(θ) represents the energy corresponding to the preset direction θ, p l,k (θ) represents the probability of the preset direction θ at the time-frequency point (l, k), z ref (l, k) represents the third ambient audio signal, E(θ1) represents the energy corresponding to the preset direction θ1, E(θ2) represents the energy corresponding to the preset direction θ2, E(θ3) represents the energy corresponding to the preset direction θ3, and E(θ N ) indicates the preset direction θ N The corresponding energy.

[0155] Optionally, since the number of sound sources included in the first ambient audio signal is unknown, when selecting a preset direction, based on the preset energy, when the energy corresponding to the preset direction is greater than the preset energy, it is determined that there is a sound source in the preset direction, and the preset direction is selected, and in this way, at least one preset direction whose corresponding energy is greater than the preset energy is selected in turn.

[0156] In the embodiment of the present disclosure, based on the spatial information contained in the multiple first ambient audio signals collected by the microphone array, signal characteristics of the multiple first ambient audio signals are obtained, and the signal characteristics can indicate the spatial information, so that the direction of the sound source is predicted based on the signal characteristics, and the direction of the sound source is obtained, thereby achieving accurate judgment of the direction of the sound source.

[0157] In the above step S102, the step of identifying a plurality of first ambient audio signals based on the direction of the sound source to obtain a sound source audio signal corresponding to each sound source can be performed as follows: Figure 4 The illustrated implementation achieves:

[0158] Figure 4 is a flowchart of another method for processing an audio signal according to an exemplary embodiment, which is executed by an electronic device, see Figure 4 , the method comprises the following steps:

[0159] Step S401: determining a fourth ambient audio signal based on a plurality of first ambient audio signals.

[0160] Among them, the fourth ambient audio signal is used to characterize multiple first ambient audio signals. The fourth ambient audio signal can be any audio signal among the multiple first ambient audio signals, or it can be an average audio signal of the multiple first ambient audio signals. The fourth ambient audio signal can be the same as or different from the above-mentioned second ambient audio signal or the third ambient audio signal.

[0161] Step S402: Based on the direction information and the direction of the sound source, identify the sound source and the sound source audio signal corresponding to each sound source from the fourth ambient audio signal.

[0162] In the embodiment of the present disclosure, after the direction of the sound source is determined, that is, the number of sound sources that need to be identified from the fourth ambient audio signal is determined, that is, the number of sound sources is determined, and when identifying the sound, the identification is performed based on the direction information.

[0163] In some embodiments, the sound source audio signal includes amplitude and phase. For each sound source, the probability of the direction of the sound source at each time-frequency point is multiplied by the amplitude of each time-frequency point in the fourth ambient audio signal to obtain the amplitude of the sound emitted by the sound source; the phase of each time-frequency point in the fourth ambient audio signal is used as the phase of the sound emitted by the sound source; based on the amplitude and phase, the sound source audio signal corresponding to the sound source is determined.

[0164] For example, the amplitude is determined by the following formula (7), and the phase is determined by the following formula (8):

[0165]

[0166]

[0167] in, represents the amplitude of the time-frequency point (l, k) in the sound emitted by the i-th sound source, |z ref (l,k)| represents the amplitude of the time-frequency point (l,k) in the fourth ambient audio signal, p l,k (θ i ) indicates the preset direction θ i The probability at the time-frequency point (l,k), Indicates the preset direction θ i The probability at the time-frequency point (l,k), ∠z ref (l, k) represents the phase of the time-frequency point (l, k) in the fourth ambient audio signal, Represents the phase of the time-frequency point (l, k) in the sound emitted by the i-th sound source.

[0168] After obtaining the phase and amplitude, the sound source audio signal corresponding to the i-th sound source can be determined as

[0169] It should be noted that the embodiments of the present disclosure are only described by taking the above-mentioned method of identifying the sound source audio signal corresponding to the sound source as an example. In another embodiment, the audio separation model can be called to process multiple first environmental audio signals to obtain the sound source audio signals corresponding to the sound source, wherein the audio separation model is used to identify the sound source audio signals corresponding to different sound sources from the audio signal. Optionally, the training process of the audio separation model includes: obtaining multiple sample audio signals and sample sound source audio signals corresponding to multiple sound sources, the multiple sample audio signals including multiple sounds, inputting the multiple sample audio signals into the audio separation model, the audio separation model outputting multiple predicted sound source audio signals, adjusting the model parameters of the audio separation model based on the differences between the multiple predicted sound source audio signals and the multiple sample sound source audio signals, and performing multiple iterative training on the audio separation model to obtain a trained audio separation model.

[0170] In the disclosed embodiments, based on the direction of the determined sound source, the sound source audio signal corresponding to the sound source is identified from the ambient audio signal, thereby achieving sound separation. For the user, the user can obtain sounds emitted by different sound sources from a variety of mixed sounds without the need for the user to distinguish them one by one, thereby improving the user experience.

[0171] The step of identifying the type of the sound source corresponding to each sound source audio signal in the above step S103 can be implemented by the following implementations:

[0172] The category recognition model is called to recognize each sound source audio signal to obtain the category of the sound source corresponding to each sound source audio signal. The category recognition model is used to recognize the category to which the sound emitted by the sound source belongs.

[0173] The category recognition model is trained based on a training data set, which includes sample sound source audio signals of multiple categories and sample categories corresponding to each sample sound source audio signal. For any category, the sample sound source audio signal of the category is the sound belonging to the category cut out from the sample audio signal, and the difference between the number of sample sound source audio signals of different categories is less than a preset value.

[0174] As for the acquisition of training data sets, it is difficult to directly obtain a large number of sample sound source audio signals of a single category. Therefore, a sample audio signal is first obtained, which may contain sounds of multiple categories. Then, the sounds belonging to the multiple categories are segmented from the sample audio signal, thereby obtaining sample sound source audio signals of more categories.

[0175] Furthermore, it is necessary to ensure that the difference between the number of sample sound source audio signals of different categories is less than a preset value, which is 0, 1, 2 or other smaller values. When the difference in the number of sample sound source audio signals of different categories is small, the training data of different categories can be balanced, so that multiple categories can be trained more evenly, thereby preventing overfitting during training of certain categories.

[0176] Optionally, for some categories with a small amount of data, data augmentation can be performed to increase the amount of data. For example, when the amount of data in category 1 is small, part of the time domain information and frequency domain information in the sample sound source audio signal 1 of the category 1 can be randomly masked (covered) to obtain the sample sound source audio signal 2 of the category 1, and part of the time domain information and frequency domain information in the sample sound source audio signal 3 of the category 1 can be randomly masked (covered) to obtain the sample sound source audio signal 4 of the category 1.

[0177] Optionally, after obtaining the sample sound source audio signal, a labeling model can be used to label the sample sound source audio signal to obtain a label corresponding to the sample sound source audio signal, and the label indicates the sample category corresponding to the sample sound source audio signal; alternatively, the sample sound can be manually labeled to obtain a label corresponding to the sample sound source audio signal; alternatively, after labeling using a labeling model, manual precision labeling can be performed, thereby ensuring the accuracy of labeling while achieving rapid labeling.

[0178] For the training process of the category recognition model, optionally, each sample sound source audio signal is input into the category recognition model, and the category recognition outputs the predicted category to which each sample sound source audio signal belongs. Based on the difference between the predicted category and the sample category to which each sample sound source audio signal belongs, the model parameters of the category recognition model are adjusted, and the trained category recognition model is obtained through multiple iterative training.

[0179] For example, the category recognition model is a transformer model. The Mel spectrum of each sample sound source audio signal is input into the category recognition model. After passing through multiple layers of Conv, Padding, Relu, Pooling, etc., multiple predicted categories are obtained. Then, the cross entropy loss function is used to calculate the loss value between the predicted category and the corresponding label, and the gradient is back-propagated and calculated. The model parameters are updated according to the gradient, and the training is iterated multiple times until the loss value is less than a certain predetermined threshold, and then the training is stopped. At the same time, the performance of the current model is calculated using the validation set. If the model performance does not improve after multiple training, the training is also stopped.

[0180] When the category recognition model is a transformer model, the transformer model includes self-attention and cross-attention mechanisms. Based on these two attention mechanisms, the similarity between audio signals of sound sources of the same category is made as large as possible, and the similarity between audio signals of sound sources of different categories is made as small as possible, so that the trained model can distinguish different categories.

[0181] For example, the model structure of the category recognition model is as follows Figure 5As shown, the Mel spectrum of the sample sound source audio signal is input into the category recognition model, and the predicted category is obtained after passing through a two-dimensional convolution layer, 6 Self-Attention networks, a normalization (layernorm) layer and a linear (Leaner) layer, where the Self-Attention network includes a Self-Attention layer, a normalization layer and two linear layers. Then, the cross entropy loss function is used to calculate the loss value between the predicted category and the corresponding label, back propagation and calculation of the gradient, and the model parameters are updated according to the gradient. The training is iterated multiple times until the loss value is less than a certain predetermined threshold, and the training is stopped. At the same time, the performance of the current model is calculated using the validation set. If the model performance does not improve after multi-training, the training is also stopped.

[0182] For example, the cross entropy loss function is as follows (9):

[0183]

[0184] Among them, L represents the loss value, L i represents the loss value corresponding to the i-th sample sound source audio signal in T sample sound source audio signals, and C represents the number of categories; ic Represents 0 or 1. If the label corresponding to the i-th sample sound source audio signal represents category c, then y ic is 1, if the label corresponding to the i-th sample sound source audio signal does not represent category c, then y ic is 0; p ic It represents the predicted probability that the category of the sound source corresponding to the i-th sample sound source audio signal belongs to category c.

[0185] Optionally, in the process of using the category recognition model, for each sound source audio signal, the output of the category recognition model is a 527-dimensional vector, each dimension corresponds to a category, and the value of each dimension represents the probability that the sound source audio signal belongs to the category corresponding to the dimension, and the category with the largest corresponding value is taken as the category of the sound source corresponding to the sound source audio signal; or, multiple values ​​are sorted, and a category is selected from the multiple categories with the largest corresponding values ​​as the category of the sound source corresponding to the sound source audio signal; or, based on the multiple categories with the largest corresponding values, the category of the sound source corresponding to the sound source audio signal is merged. For example, a sound source audio signal identifies the three categories with the largest corresponding values ​​as "dog sound", "dog barking" and "angry dog ​​barking", then after merging, the corresponding category is determined to be "dog barking".

[0186] In some embodiments, in the process of training the category recognition model, multiple sample sound source audio signals can be input simultaneously each time training, and then the predicted category of the sound source corresponding to each sound source audio signal can be output simultaneously. In the process of using the category recognition model, multiple sound source audio signals can be input simultaneously, and then the predicted category of the sound source corresponding to each sound source audio signal can be output simultaneously. Alternatively, one sound source audio signal can be input, and then the predicted category of the sound source corresponding to the sound source audio signal can be output. Alternatively, in the process of training the category recognition model, one sample sound source audio signal can be input each time training, and then the predicted category of the sound source corresponding to the sample sound source audio signal can be output. In the process of using the category recognition model, one sound source audio signal can also be input, and then the predicted category of the sound source corresponding to the sound source audio signal can be output.

[0187] In some embodiments, in the process of training the category recognition model, the Mel spectrum of each sample sound source audio signal is extracted, and the Mel spectrum of each sample sound source audio signal is input into the category recognition model. Correspondingly, in the process of using the category recognition model, the Mel spectrum of each sound source audio signal is first extracted, and the Mel spectrum of each sound source audio signal is input into the category recognition model.

[0188] The process of extracting the Mel spectrum of each sound source audio signal includes: determining the sound source audio signal, pre-emphasizing the sound source audio signal, framing, windowing and Fast Fourier Transform (FFT) the pre-emphasized sound source audio signal, filtering the sound source audio signal after FFT through a Mel filter group, and finally performing logarithmic operation on the filtered sound source audio signal to obtain the Mel spectrum. For example, the Mel spectrum is calculated using the following formula (10):

[0189]

[0190] Among them, Mel(f) represents the Mel spectrum corresponding to the sound source audio signal, and f represents the sound source audio signal after filtering.

[0191] In the embodiment of the present disclosure, the sound source is classified by identifying the category of the sound source. In addition, when training the category recognition model, the training is performed based on samples of 527 categories, so that the trained category recognition model can recognize more types of sounds and recognize more accurately.

[0192] The step of displaying the prompt information of the sound source in the display interface according to the direction of the sound source and the type of the sound source in the above step S104 can be implemented by the following implementations:

[0193] In some embodiments, the prompt information is a sound source category icon, and the sound source category icon corresponding to the sound source is displayed in the display interface. Among them, the sound source category icons corresponding to different categories of sound are different. For example, the sound source category icon corresponding to the first category is a red circle, and the sound source category icon corresponding to the second category is a green circle. Then, when there is sound 1 of the first category, a red circle is displayed in the display interface, and when there is sound 2 of the second category, a green circle is displayed in the display interface. The position of the sound source category icon in the display interface indicates the direction of the sound source. For example, taking the position representing the electronic device in the display interface as the center, the sound source category icon is displayed in the upper right corner of the display interface, and it is determined that the sound source comes from the right front of the electronic device. For example, see Figure 6 The radar chart shown in the figure displays a sound source category icon, which indicates the direction of the cat's meow and the sound source of the cat's meow.

[0194] Optionally, there are reference points and reference lines in the display interface, the reference point represents the position of the electronic device, and the reference line represents 0°. In this case, the sound source category icon is displayed in the display interface according to the reference line. For example, if the electronic device locates the direction of the sound source at 15°, the sound source category icon corresponding to the sound source is displayed in the display interface in the direction of 15° with the reference line at 0° as the standard.

[0195] It should be noted that the embodiment of the present disclosure only identifies the direction of the sound source. In another embodiment, the distance between the sound source and the electronic device is identified. In this case, the position of the sound source category icon in the display interface indicates the direction of the sound source and the distance between the sound source and the electronic device.

[0196] In some embodiments, the prompt information is text information, and the text information corresponding to the sound source is displayed in the display interface, wherein the text information corresponding to the sound source is used to describe the direction of the sound source and the category of the sound emitted by the sound source.

[0197] In some embodiments, the prompt information is a sound source category icon and text information, and the text information corresponding to the sound source is displayed next to the sound source category icon corresponding to the sound source in the display interface.

[0198] In some embodiments, the display interface includes a coordinate graph, and the prompt information includes a sound source category icon; displaying the prompt information of the sound source in the display interface includes: determining a sound source category icon corresponding to the category of the searched sound source according to the category of the sound source; determining a target coordinate position according to the direction of the sound source; and displaying the sound source category icon at the target coordinate position on the coordinate graph. The target position may be at least one coordinate in the coordinate graph.

[0199] For the display interface, optionally, the user triggers a preset operation in the electronic device, and the electronic device displays the display interface, for example, the predicted operation is an operation of opening a preset application.

[0200] In the embodiment of the present disclosure, after the direction and category of the sound source are identified, the direction and category of the sound source are displayed through a display interface. The direction and category of the sound source are intuitively presented to the user through the display interface, so that the user can understand the sound conditions in the surrounding environment even if he cannot hear the sound, thereby improving the user experience.

[0201] The audio signal processing process shown in the above implementation can be expressed as Figure 7 See the schematic diagram shown in Figure 7 First, the microphone signal is obtained (i.e., multiple first ambient audio signals collected by multiple microphones are obtained), and then the sound source audio signals corresponding to each sound source in the multiple first ambient audio signals are located and separated to obtain the sound source audio signals 1-N and the DOA corresponding to each sound source, and then the sound event detection is performed on the sound source audio signals 1-N (i.e., the category of the sound emitted by the sound source is identified) to obtain the event category 1-N corresponding to the sound source audio signals 1-N. Finally, according to the DOA corresponding to the sound source audio signals 1-N and the corresponding event category 1-N, the DOA and event category corresponding to each sound source audio signal are displayed on the front end. Among them, the implementation methods of locating the sound source and separating the sound source audio signals, identifying the event category and displaying refer to the above embodiments.

[0202] The audio signal processing method provided by the embodiments of the present disclosure can be applied in various scenarios.

[0203] For example, in the scenario of helping hearing-impaired people feel the sounds of the surrounding environment, a hearing-impaired person walks in the park wearing a smart watch, which has the function of "determining the direction and category of the sound source" and a display screen in the smart watch. When he / she walks to the center of the park, he / she suddenly perceives some sounds, but he / she cannot determine the source and category of the sound source, so he / she opens the radar chart on the smart watch and finds that several different icons appear on the display screen, each icon represents a sound event (i.e., the sound source audio signal), and the position of the icon in the radar chart indicates the direction of the sound source. Through this function, he / she can better understand the sound conditions in the surrounding environment, so as to better integrate into society and enjoy life.

[0204] For another example, in a security scenario, an electronic device with the function of "determining the direction and category of a sound source" is installed at a fixed location in a situation where security protection is required. The electronic device uses the above-mentioned implementation method to determine the source and category of the sound source, and then displays the source and category of each sound source in a radar chart, or sends the source and category of each sound source to other electronic devices for display. Security personnel can monitor in real time whether there are safety hazards based on the source and category of the sound source displayed by the electronic device.

[0205] Figure 8 is a block diagram of an audio signal processing device according to an exemplary embodiment, which is configured in an electronic device, see Figure 8 , the device comprises:

[0206] The signal acquisition module 801 is configured to acquire a plurality of first ambient audio signals, each of which includes a sound emitted by at least one sound source, and the plurality of first ambient audio signals are collected by different microphones;

[0207] A direction determination module 802 is configured to determine the direction of each sound source based on a plurality of first ambient audio signals;

[0208] The signal recognition module 803 is configured to recognize multiple first ambient audio signals based on the direction of the sound source, and obtain a sound source audio signal corresponding to each sound source, wherein each sound source audio signal includes a sound emitted by the sound source corresponding to the direction of the sound source;

[0209] A category identification module 804 is configured to identify the category of the sound source corresponding to each sound source audio signal;

[0210] The display module 805 is configured to display prompt information of the sound source in the display interface according to the direction of the sound source and the category of the sound source. The prompt information is used to represent the direction of each sound source and the category to which the sound emitted by the sound source belongs.

[0211] In some embodiments, the direction determination module 802 is configured to:

[0212] Acquire signal features of multiple first ambient audio signals, where the signal features are used to characterize multiple time-frequency points and phase information of each first ambient audio signal, and multiple time-frequency points in the multiple first ambient audio signals are the same;

[0213] The direction of the sound source is predicted based on the signal characteristics to determine the direction information, where the direction information includes the probability of each of the multiple preset directions at the multiple time-frequency points, and the probability of the preset direction at the time-frequency point indicates the possibility that the preset direction is the direction of the time-frequency point;

[0214] Based on the directional information, the direction of each sound source is determined.

[0215] In some embodiments, the direction determination module 802 is configured to:

[0216] Determine a plurality of time-frequency points based on the number of time frames and the number of frequency points of any first ambient audio signal;

[0217] Selecting any first ambient audio signal from a plurality of first ambient audio signals as a second ambient audio signal;

[0218] According to the relationship between each first ambient audio signal and the second ambient audio signal at each time-frequency point, the phase information of each first ambient audio signal at each time-frequency point is determined.

[0219] In some embodiments, the direction determination module 802 is configured to:

[0220] For each first ambient audio signal, determining a ratio of the first ambient audio signal to the second ambient audio signal at each time-frequency point, where the ratio of the time-frequency point is used to characterize the phase of the first ambient audio signal at the time-frequency point;

[0221] The phase corresponding to the ratio of each time-frequency point is determined, and the phases of the multiple time-frequency points are used as phase information.

[0222] In some embodiments, the direction determination module 802 is configured to:

[0223] The sound source localization model is called to predict the direction of the sound source based on the signal characteristics to obtain direction information. The sound source localization model is used to predict the direction of any sound source.

[0224] In some embodiments, the direction determination module 802 is configured to:

[0225] Determine a third ambient audio signal based on the multiple first ambient audio signals, where the third ambient audio signal is used to characterize the multiple first ambient audio signals;

[0226] For each preset direction, based on the probability of the preset direction at multiple time-frequency points and the third ambient audio signal, determine the energy corresponding to the preset direction, the energy corresponding to the preset direction being used to characterize the possibility that the preset direction is the direction of any sound source;

[0227] Based on energies corresponding to a plurality of preset directions, the direction of each sound source is determined, and the energy corresponding to the direction of each sound source is greater than the energy corresponding to other preset directions.

[0228] In some embodiments, the direction determination module 802 is configured to:

[0229] Determine a fourth ambient audio signal based on the plurality of first ambient audio signals, where the fourth ambient audio signal is used to characterize the plurality of first ambient audio signals;

[0230] Based on the direction information and the direction of the sound source, a sound source audio signal corresponding to each sound source is identified from the fourth ambient audio signal.

[0231] In some embodiments, the direction determination module 802 is configured to:

[0232] For each sound source, multiply the probability of the direction of the sound source at each time-frequency point by the amplitude of each time-frequency point in the fourth ambient audio signal to obtain the amplitude of the sound emitted by the sound source;

[0233] Using the phase of each time-frequency point in the fourth ambient audio signal as the phase of the sound emitted by the sound source;

[0234] Based on the amplitude and phase, a sound source audio signal corresponding to the sound source is determined.

[0235] In some embodiments, the category identification module 804 is configured to:

[0236] A category recognition model is called to recognize each sound source audio signal to obtain the category of the sound source corresponding to each sound source audio signal, wherein the category recognition model is trained based on a training data set, the training data set includes sample sound source audio signals of multiple categories and a sample category corresponding to each sample sound source audio signal, the sample sound source audio signal of a category is a sound source audio signal belonging to a category cut out from the sample audio signal, and the difference between the numbers of sample sound source audio signals of different categories is less than a preset value.

[0237] In some embodiments, the prompt information is a sound source category icon, and the display module 805 is configured to:

[0238] The sound source category icon corresponding to the sound source is displayed in the display interface, wherein different categories of sound sources correspond to different sound source category icons, and the position of the sound source category icon in the display interface indicates the direction of the sound source.

[0239] In some embodiments, the display interface includes a coordinate graph, and the prompt information includes a sound source category icon; the display module 805 is configured to:

[0240] determining a sound source category icon corresponding to the category of the searched sound source according to the category of the sound source;

[0241] Determine the target coordinate position according to the direction of the sound source;

[0242] The sound source category icon is displayed at the target coordinate position on the coordinate graph.

[0243] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0244] An embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the audio signal processing method in the above embodiment.

[0245] Fig. 9 is a block diagram of an electronic device 900 according to an exemplary embodiment.

[0246] Reference Fig. 9, the electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0247] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0248] The memory 904 is configured to store various types of data to support operations on the electronic device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0249] The power supply component 906 provides power to the various components of the electronic device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.

[0250] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0251] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0252] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0253] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the electronic device 900, the relative positioning of components, such as the display and keypad of the electronic device 900, and the sensor assembly 914 can also detect the position change of the electronic device 900 or a component of the electronic device 900, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and the temperature change of the electronic device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0254] The communication component 916 is configured to facilitate wired or wireless communication between the electronic device 900 and other devices. The electronic device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0255] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0256] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by a processor 920 of an electronic device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0257] The embodiments of the present disclosure also provide a non-transitory computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the audio signal processing method in the above embodiment.

[0258] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0259] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for processing an audio signal, characterized in that: The method comprises: Acquire multiple first ambient audio signals, each of which includes a sound emitted by at least one sound source, and the multiple first ambient audio signals are collected by different microphones; Determining the direction of each of the sound sources based on the plurality of first ambient audio signals; Identify the plurality of first ambient audio signals based on the direction of the sound source to obtain a sound source audio signal corresponding to each of the sound sources, wherein each sound source audio signal includes a sound emitted by a sound source corresponding to the direction of the sound source; Identifying the category of the sound source corresponding to each of the sound source audio signals; Prompt information of the sound source is displayed in a display interface according to the direction of the sound source and the category of the sound source, wherein the prompt information is used to characterize the direction of each sound source and the category to which the sound emitted by the sound source belongs.

2. The method according to claim 1, characterized in that The determining the direction of each of the sound sources based on the plurality of first ambient audio signals comprises: Acquire signal characteristics of the multiple first ambient audio signals, where the signal characteristics are used to characterize multiple time-frequency points and phase information of each of the first ambient audio signals, and multiple time-frequency points in the multiple first ambient audio signals are the same; Predicting the direction of the sound source based on the signal characteristics to determine direction information, the direction information including the probability of each of the preset directions at the multiple time-frequency points, the probability of the preset direction at the time-frequency point indicating the possibility that the preset direction is the direction of the time-frequency point; Based on the direction information, the direction of each of the sound sources is determined.

3. The method according to claim 2, characterized in that The obtaining signal characteristics of the plurality of first ambient audio signals comprises: Determine the multiple time-frequency points based on the number of time frames and the number of frequency points of any one of the first ambient audio signals; Selecting any one of the plurality of first ambient audio signals as the second ambient audio signal; The phase information of each of the first ambient audio signals at each of the time-frequency points is determined according to the relationship between each of the first ambient audio signals and the second ambient audio signals at each of the time-frequency points.

4. The method according to claim 3, characterized in that The determining, according to the relationship between each of the first ambient audio signals and the second ambient audio signals at each of the time-frequency points, the phase information of each of the first ambient audio signals at each of the time-frequency points comprises: For each of the first ambient audio signals, determining a ratio of the first ambient audio signal to the second ambient audio signal at each of the time-frequency points, wherein the ratio of the time-frequency points is used to characterize a phase of the first ambient audio signal at the time-frequency point; A phase corresponding to the ratio of each of the time-frequency points is determined, and the phases of the multiple time-frequency points are used as the phase information.

5. The method according to claim 2, characterized in that: The predicting of the direction of the sound source based on the signal feature to determine the direction information includes: The sound source localization model is called to predict the direction of the sound source for the signal feature to obtain the direction information. The sound source localization model is used to predict the direction of any sound source.

6. The method according to claim 2, characterized in that The determining the direction of each of the sound sources based on the direction information comprises: Determine a third ambient audio signal based on the multiple first ambient audio signals, where the third ambient audio signal is used to represent the multiple first ambient audio signals; For each of the preset directions, based on the probability of the preset direction at the multiple time-frequency points and the third ambient audio signal, determining the energy corresponding to the preset direction, wherein the energy corresponding to the preset direction is used to characterize the possibility that the preset direction is the direction of any sound source; Based on the energies corresponding to the multiple preset directions, the direction of each of the sound sources is determined, and the energy corresponding to the direction of each of the sound sources is greater than the energies corresponding to other preset directions.

7. The method according to claim 2, characterized in that: The identifying the plurality of first ambient audio signals based on the direction of the sound source to obtain a sound source audio signal corresponding to each of the sound sources includes: Based on the multiple first ambient audio signals, determine a fourth ambient audio signal, where the fourth ambient audio signal is used to characterize the multiple first ambient audio signals; Based on the direction information and the direction of the sound source, a sound source audio signal corresponding to each of the sound sources is identified from the fourth ambient audio signal.

8. The method according to claim 7, characterized in that The step of identifying a sound source audio signal corresponding to each of the sound sources from the fourth ambient audio signal based on the direction information and the direction of the sound source comprises: For each sound source, multiply the probability of the direction of the sound source at each time-frequency point by the amplitude of each time-frequency point in the fourth ambient audio signal to obtain the amplitude of the sound emitted by the sound source; Using the phase of each of the time-frequency points in the fourth ambient audio signal as the phase of the sound emitted by the sound source; Based on the amplitude and the phase, a sound source audio signal corresponding to the sound source is determined.

9. The method according to claim 1, characterized in that: The identifying the category of the sound source corresponding to each of the sound source audio signals comprises: Calling a category recognition model to recognize each of the sound source audio signals and obtain the category of the sound source corresponding to each of the sound source audio signals, wherein the category recognition model is trained based on a training data set, the training data set includes sample sound source audio signals of multiple categories and a sample category corresponding to each of the sample sound source audio signals, the sample sound source audio signals of the category are sound source audio signals belonging to the category cut out from the sample audio signals, and the difference between the numbers of sample sound source audio signals of different categories is less than a preset value.

10. The method according to any one of claims 1 to 9, characterized in that: The prompt information is a sound source category icon, and the prompt information of the sound source is displayed in the display interface, including: A sound source category icon corresponding to the sound source is displayed in the display interface, wherein different categories of sound sources correspond to different sound source category icons, and the position of the sound source category icon in the display interface indicates the direction of the sound source.

11. The method according to any one of claims 1 to 9, characterized in that: The display interface includes a coordinate diagram, and the prompt information includes a sound source category icon; and the prompt information of the sound source is displayed in the display interface, including: Determining a sound source category icon corresponding to the category of the search sound source according to the category of the sound source; Determine the target coordinate position according to the direction of the sound source; The sound source category icon is displayed at the target coordinate position on the coordinate graph.

12. An audio signal processing device, characterized in that: The device comprises: a signal acquisition module, configured to acquire a plurality of first ambient audio signals, each of which includes a sound emitted by at least one sound source, and the plurality of first ambient audio signals are collected by different microphones; a direction determination module, configured to determine the direction of each of the sound sources based on the plurality of first ambient audio signals; a signal recognition module configured to recognize the plurality of first ambient audio signals based on the direction of the sound source, and obtain a sound source audio signal corresponding to each of the sound sources, wherein each sound source audio signal includes a sound emitted by a sound source corresponding to the direction of the sound source; A category identification module, configured to identify the category of the sound source corresponding to each of the sound source audio signals; The display module is configured to display prompt information of the sound source in a display interface according to the direction of the sound source and the category of the sound source, wherein the prompt information is used to characterize the direction of each sound source and the category to which the sound emitted by the sound source belongs.

13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method as claimed in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 11.

Images