Voiceprint separation method of intelligent glasses, electronic equipment and storage medium

Abstract

The invention relates to a voiceprint separation method of intelligent glasses, electronic equipment and a storage medium. The method comprises the following steps: performing time synchronization processing on target mixed audio data collected by a microphone on the intelligent glasses and target image data of a recognition object collected by a camera; obtaining a first feature and a second feature of the target mixed audio data and a third feature of the target image data, the first feature being a basic attribute feature of the target mixed audio; and inputting the first feature into a voiceprint separation neural network model, and fusing the second feature and the third feature in the voiceprint separation neural network model so as to separate the sound of the recognition object from the target mixed audio data. The voiceprint separation method and device solve the technical problem that voiceprint separation methods in related technologies are generally low in accuracy.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a voiceprint separation method, electronic device and storage medium for smart glasses. Background Art

[0002] Smart glasses, also known as smart mirrors or wearable computing glasses, are glasses that integrate microcomputers and advanced electronic technology, designed to provide functions similar to smartphones or tablets to display information directly in the user's field of view. Smart glasses are used in more and more occasions, one of which is the translation function, which recognizes the speaker's speech, translates it into the wearer's native language, and displays it on the glasses screen, allowing the wearer to communicate easily with foreigners.

[0003] However, the voiceprint separation methods in the related art generally have the technical problem of low accuracy, especially in the scene where multiple people speak at the same time, the differentiation is not ideal. For this technical problem, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a voiceprint separation method, an electronic device, and a storage medium for smart glasses, so as to at least solve the technical problem of low accuracy that is common in voiceprint separation methods in related technologies.

[0005] According to one aspect of an embodiment of the present application, a voiceprint separation method for smart glasses is provided, comprising: performing time synchronization processing on target mixed audio data collected by a microphone on the smart glasses and target image data of an identified object collected by a camera; acquiring a first feature and a second feature of the target mixed audio data, and a third feature of the target image data, wherein the first feature is a basic attribute feature of the target mixed audio; inputting the first feature into a voiceprint separation neural network model, and fusing the second feature and the third feature in the voiceprint separation neural network model to separate the sound of the identified object from the target mixed audio data.

[0006] Optionally, the second feature includes at least one of the following: arrival direction of the sound source signal, arrival time of the sound source signal, and energy of the sound source signal.

[0007] Optionally, the third feature includes the position of an anchor box with the highest confidence in the target image data and the confidence of the anchor box.

[0008] Optionally, the voiceprint separation neural network model includes an encoding network and a decoding network, the encoding network and the decoding network include multiple network blocks, and the decoding network also includes a fusion module, and the fusion module is used to fuse the second feature and the third feature.

[0009] Optionally, the fusion module is located in the last network block of the decoding network.

[0010] Optionally, the network block includes multiple network layers, the first layer of the multiple network layers is a convolutional neural network CNN, the second layer is a memory processing unit, and the third layer is CNN, the memory processing unit includes a recurrent neural network RNN ​​or a long short-term memory neural network LSTM or a merging processing module with a memory unit, the memory unit is used to store historical features, the merging processing module is used to merge the historical features with the current features, and update the current features to the memory unit.

[0011] Optionally, the fusion module is located before the memory processing unit.

[0012] Optionally, time synchronization processing of target mixed audio data collected by the microphone on the smart glasses and target image data of the identified object collected by the camera includes: obtaining the time difference between audio transmission and light transmission from the identified object; the time difference is used to forward correct the collection time of the target mixed audio data.

[0013] Optionally, the time difference is obtained according to the quotient of the estimated distance corresponding to the current application scenario and the sound propagation speed.

[0014] Optionally, the smart glasses include a microphone array consisting of multiple microphones, and obtain the time difference between the audio transmission and the light transmission from the identified object, including: determining the time when the target mixed audio data arrives at each microphone respectively; calculating the position distance of the identified object according to the time when the target mixed audio data arrives at each microphone respectively and the array shape of the microphone array; calculating the time difference according to the quotient of the position distance and the speed of sound propagation.

[0015] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a processor, and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method in any of the above embodiments.

[0016] According to another aspect of the embodiments of the present application, a non-transitory machine-readable medium storing computer instructions is also provided, where the computer instructions are used to enable a computer to execute the method in any of the above embodiments.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which, when executed by a processor of a computer, causes the computer to execute the method in any of the above embodiments.

[0018] Beneficial effects of the embodiments of the present application:

[0019] In the embodiment of the present application, in view of the different acquisition delays of sound and video, and the relatively large sound delay, the target mixed audio data collected by the microphone on the smart glasses and the target image data of the identified object collected by the camera are time-synchronized. In addition, when performing voiceprint separation through the voiceprint separation neural network model, in addition to using the characteristics of the target mixed audio data itself (the first characteristic), the second characteristic of the target mixed audio data and the third characteristic of the target image data are also integrated, which improves the accuracy of voiceprint separation, thereby solving the technical problem of low accuracy that is common in voiceprint separation methods in related technologies.

[0020] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a voiceprint separation method for smart glasses according to an embodiment of the present application;

[0023] Figure 2a is a position distribution diagram of microphones on smart glasses according to an embodiment of the present application;

[0024] Figure 2b is a location distribution diagram of microphones on another smart glasses according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of echo cancellation according to an embodiment of the present application;

[0026] Figure 4 is another schematic diagram of echo cancellation according to an embodiment of the present application;

[0027] Figure 5 is a flow chart of a silence detection method according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of camera positions according to an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of a generalized cross-correlation delay estimation method according to an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of a method for estimating far-field sound source positions of two microphones according to an embodiment of the present application;

[0031] Fig. 9 is a schematic diagram of a near-field sound source localization estimation method using three microphones according to an embodiment of the present application;

[0032] Fig.10 is a flow chart of a first feature acquisition method according to an embodiment of the present application;

[0033] Fig.11 is a schematic diagram of a YOLO parameter list according to an embodiment of the present application;

[0034] Fig.12 is another schematic diagram of a YOLO parameter list according to an embodiment of the present application;

[0035] Fig.13 This is a diagram of the architecture of a neural network model for voiceprint separation according to an embodiment of the present application;

[0036] Fig.14 is a schematic diagram of a network block structure according to an embodiment of the present application;

[0037] Fig.15 is a schematic diagram of another structure of a voiceprint separation neural network model according to an embodiment of the present application;

[0038] Fig.16 is a schematic diagram of the structure of another voiceprint separation neural network model according to an embodiment of the present application;

[0039] Fig.17 is another schematic diagram of a network block structure according to an embodiment of the present application;

[0040] Fig.18 is a schematic diagram of the structure of another voiceprint separation neural network model according to an embodiment of the present application;

[0041] Fig.19 is a structural schematic diagram of a voiceprint separation device for smart glasses according to an embodiment of the present application;

[0042] Fig. 20 Schematic diagram of the structure of the electronic device of this embodiment. DETAILED DESCRIPTION

[0043] Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

[0044] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0045] CPU, Central Processing Unit, also called general-purpose processor, is the "brain" of smart devices.

[0046] DSP, Digital Signal Processing, digital signal processing.

[0047] ASR, Automatic Speech Recognition, speech recognition.

[0048] AR, Augmented Reality.

[0049] VAD, voice activity detection, voice endpoint detection, aims to detect whether there is a voice signal.

[0050] Speech, human voice.

[0051] Noise, non-human voice, or noise.

[0052] YOLO, "You Only Look Once" is an algorithm for object detection using convolutional neural networks.

[0053] The object detection anchor box describes the coordinates of the four points of the rectangle that contains the object of interest in the image.

[0054] MFCCs, Mel Frequency Cepstral Coefficents, are cepstral parameters extracted in the Mel scale frequency domain and are a feature widely used in automatic speech and speaker recognition.

[0055] FBANK, filter bank, an audio feature.

[0056] RNN, Recurrent Neural Network, recurrent neural network.

[0057] LSTM, Long Short-Term Memory, Long Short-Term Memory Neural Network

[0058] GRU, gated recurrent unit, functions similarly to LSTM.

[0059] In the related art, the voiceprint separation neural network model only uses the characteristic information of the audio itself for voiceprint separation, resulting in low voiceprint separation accuracy. In order to solve this problem, the embodiments of the present application provide relevant solutions, which are described in detail below. Among them, the application scenarios of the embodiments of the present application include but are not limited to: using smart glasses for translation, using smart glasses in conference scenarios to separate and store the speech content of the main speaker. That is, in an environment with a lot of background noise, the present application can be used in any application scenario where the target sound needs to be separated.

[0060] Figure 1 is a flow chart of a voiceprint separation method for smart glasses according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0061] Step S102, performing time synchronization processing on the target mixed audio data collected by the microphone on the smart glasses and the target image data of the recognized object collected by the camera.

[0062] It should be noted that the target mixed audio data may include audio data from multiple different sound sources, such as audio data of the wearer of the smart glasses, audio data of the non-wearer of the smart glasses, audio data of the talker, ambient audio data, etc. The target image data of the identified object may be image data of the speaker closest to the wearer of the smart glasses, or an image captured by the camera of the smart glasses. It is understandable that the camera field of the smart glasses is usually directly in front of the wearer, and the wearer often faces the identified object, so the identified object will be within the camera field of view, and the target image captured by the camera can include the identified object, so that the target image can be used in the voiceprint separation process for the identified object.

[0063] In some embodiments of the present application, the microphone may include multiple microphones, for example, forming a microphone array. When there are two microphones, the present application embodiment provides the following: Figure 2a-2bThe position distribution diagram of the microphones on the smart glasses shown in the figure. The microphones on the smart glasses continuously record, and transmit the collected mixed audio data to the wake-up module ("wake-up module" refers to a combination of hardware and software that can enable smart glasses to quickly activate from sleep or standby state and start responding to user commands) for processing every period of time (one frame). The mixed audio data generally includes three channels: microphone 1, the audio collected by the microphone close to the wearer's mouth. Since microphone 1 is closer to the wearer's mouth, the energy of the wearer's audio collected by it is stronger than that of other microphones; microphone 2, the audio collected by the microphone far away from the wearer's mouth. Compared with microphone 1, the energy of the wearer's audio collected by it is lower, but it is more suitable for recording the environment and non-wearer recording; recording of the sound played by the smart glasses speaker. In the process of voice interaction, the signal formed after the sound played by the speaker is collected by the microphone is called echo, and the purpose of the microphone is to collect the human voice. The echo needs to be eliminated, otherwise it will interfere with the microphone's collection of human voice and speech recognition. Therefore, the above mixed audio data can be echo-cancelled to determine the above target mixed audio data. In one possible implementation, the embodiment of the present application provides the following Figure 3 , Figure 4 There are two echo cancellation methods shown in Figure 1. Figure 3 , 4 As shown, echo cancellation estimates the delay of the echo signal based on the signal played by the speaker (text-to-speech (TTS) signal) and the signal collected by the recording. The delay estimation method mainly uses Kalman filtering. After the delay is obtained, the echo signal and the recording signal are aligned according to the delay, and the echo signal is subtracted from the recording data. For example, a filter is designed, and its output signal is the same as the echo waveform, but the phase is opposite, and it is added to the recording data to cancel the echo signal. The main filter algorithm is Normalized Least Mean Square (NLMS). In actual use, the echo residual caused by nonlinearity must also be considered, and an additional module is required to eliminate the echo residual. Echo cancellation algorithms based on neural networks can also be used. Low-power echo cancellation means that the echo cancellation runs on low-power equipment with relatively small computing power. If it is based on DSP, a fixed-point program is generally used. If it is based on neural networks, the model size is generally small.

[0064] In addition, the dual microphones need to perform silence detection on the audio after echo cancellation of each microphone. During the voice interaction process, the user is silent for more than half of the time, and more than half of the audio signals collected by the recording are silent. Sending these silent audios to voice recognition for processing wastes computing resources. If they are passed to the cloud for recognition, it also wastes network bandwidth. Therefore, it is necessary to detect the silent part, delete the silence, and then hand it over to the subsequent processing module. The silence detection process is similar to wake-up. In one possible implementation, the embodiment of the present application provides Figure 5 The flow chart of the silence detection method is shown in FIG. Figure 5 As shown, it includes: extracting features from audio data; determining an acoustic model; when using a neural network as an acoustic model, the fully connected layer multiplies the input feature vector by the weight matrix and adds a bias to form an output vector; softmax / sigmoid is usually used in the last layer of the classification task to convert the output vector of the fully connected layer into a probability distribution; the probability output by the softmax / sigmoid layer represents the confidence of each category. In the Voice Activity Detection (VAD) system, it is usually necessary to set a threshold to determine whether the current audio segment belongs to speech or silence. Double talk means that both parties are talking. The simplest method is to perform silence detection based on the signals of the two microphones. If a human voice is detected, it means that the person corresponding to the microphone is talking. The first microphone MIC1 corresponds to the wearer, and the second microphone MIC2 corresponds to the non-wearer / talker / external environment. Among them, the results of the double talk judgment may include: no one is talking; only the wearer is talking; only the non-wearer is talking; both the wearer and the non-wearer are talking. When it is detected that both the wearer and the non-wearer are speaking, the above-mentioned voiceprint separation step can be performed. If it is detected that no one is speaking, it directly returns to continue collecting the next frame of data.

[0065] In some embodiments of the present application, if the wearer and the non-wearer are speaking at the same time, beamforming is also required to enhance the signal of the non-wearer, such as the signal collected by MIC2 in the above example. In the far-field voice interaction scenario, especially when there is environmental interference, the quality of the collected signal is poor and the subsequent recognition is difficult, so a microphone array is formed by multiple microphones, and the spatial characteristics of the signals collected by the multiple microphones are used to enhance the target direction signal and suppress the non-target direction signal to improve the signal-to-noise ratio. The simplest way to enhance speech with a microphone array is to first find the target direction (the microphone in the 0-degree direction of the front, directly in front of the wearer), calculate the delay between other microphones and the front microphone, and then sum the delays of all microphone signals to achieve the purpose of enhancement. In actual applications, the sound signal source and any microphone are not necessarily in a 0-degree relationship in the front, and all microphones receive the same signal at different times and with different phases. It is necessary to use a beamforming method to enhance one or all microphone signals. The generalized sidelobe cancellation (GSC) algorithm can be used with post-filtering. Smart glasses have two microphones. The first one is close to the wearer's mouth. Since microphone 1 is closer to the wearer's mouth, it can collect stronger audio energy than other microphones. The second one is far away from the wearer's mouth. Compared with microphone 1, it can collect lower audio energy, but is more suitable for recording ambient and non-wearer audio. Beamforming is to use the microphone array formed by these two microphones to enhance one of the signals. For example, in the translation scenario, the voice of the non-wearer is enhanced.

[0066] In some embodiments of the present application, according to the result of the double-talk judgment, the signal to be identified is transmitted to the mobile phone through the Bluetooth connection between the smart glasses and the terminal (mobile phone, tablet computer, etc.) for further processing, as shown in Table 1.

[0067] Table 1

[0068] Double-talk judgment result Transfer content No one is talking Do not transfer Only the wearer speaks Do not transfer Only non-wearers are talking Transmit the second signal Wearers and non-wearers alike are talking Transmitting mixed signals after beamforming

[0069] In some embodiments of the present application, when the camera collects target image data of the identified object, at least one camera can be deployed on the smart glasses, facing the front, in the middle or on the left and right sides. In a possible implementation, a Figure 6The camera position diagram shown in the figure is a method of placing the camera in the middle. The purpose of taking pictures is to collect the lip images of the person on the other side, and use the lip photos taken before and after to determine whether the person on the other side is speaking. Taking pictures is generally only triggered in the translation scene, and 2 to 8 frames of pictures are collected per second. The smaller the collection frequency, the less calculation is required, but too little may lead to misjudgment. The size and pixels of the photos taken in the translation scene are fixed. For example, the photo size is 480*600, which is consistent with the sample data used in training.

[0070] After acquiring the target mixed audio data and the target image data, since the acquisition delays of sound and image are different, and the sound delay is relatively large, the embodiment of the present application proposes a step of time synchronization processing of the target mixed audio data and the target image data. This step includes: acquiring the time difference between the audio transmission from the identified object and the light transmission (light transmission is the process of light passing through the camera lens from the shooting scene and finally reaching the image sensor), and the time difference is used to correct the acquisition time of the target mixed audio data forward.

[0071] For example, the calculated time difference is Δt. Since the audio signal lags behind the image, in the subsequent voiceprint separation process, the audio data time can be advanced by Δt based on the time difference Δt, so that the audio data and video data can be synchronized. For example, the target image data 1 and the target mixed audio data 1 are collected at time point t1. Since the speed of light propagates very fast, the target image data 1 can be considered as the actual image at time point t1, but the speed of sound is much slower than the speed of light, so the target mixed audio data 1 is not actually the audio data generated at time point t1. The audio data generated at time point t1 should be the target mixed audio data 2 collected at time point t1+Δt. In view of this, the present application advances the target mixed audio data 2 to time point t1, that is, performs time synchronization processing with the target image data 1, so that the target mixed audio data 2 and the target image data 1 can be combined for subsequent processing, which is conducive to improving the accuracy of voiceprint separation.

[0072] The above-mentioned time difference determination method may include: method 1, estimation based on empirical values. Method 2, sound source positioning based on multiple microphones. In some embodiments of the present application, the above-mentioned method 1 may be obtained based on the quotient of the estimated distance corresponding to the current application scenario and the sound propagation speed. For example, the translation scenario is interested in the voice of the person talking opposite the glasses wearer. Usually the person talking stands about 1 meter away from the glasses wearer. The sound transmission delay of 1 meter is: 1000 / 340=2.94 milliseconds, that is, the audio arrives about 2.94ms slower than the video. This 2.94ms delay needs to be compensated to the audio and video data. Assuming that the audio is collected at a frame rate of 10ms, a sampling rate of 16000, and a bit width of 16bit, each frame of data contains 16000 sampled data. The delay of 2.94ms needs to correspond to 4704 sampled points of the sampled data, that is, the current frame data needs to be moved forward by 4704 sampled points to be synchronized with the currently taken photo.

[0073] The above-mentioned method 2 includes: determining the time when the target mixed audio data arrives at each microphone respectively; calculating the position distance of the identification object according to the time when the target mixed audio data arrives at each microphone respectively, and the array shape of the microphone array; calculating the time difference according to the quotient of the position distance and the sound propagation speed. For example, taking the audio and video synchronization compensation of 3 or more microphone glasses as an example, the audio and video time synchronization compensation described in the above method 1 is roughly estimated based on empirical values. Since the talker moves the position and orientation at any time, the estimation of the delay is not very accurate. The embodiment of the present application proposes a reliable method for dynamically detecting the distance of the talker, that is, sound source positioning based on multiple microphones. For example, the Time Difference of Arrival (TDOA) method, the sound source positioning method based on TDOA is generally divided into two steps: Step 1, calculate the time difference (delay estimation) of the sound source signal arriving at the microphone array; Step 2, establish a sound source positioning model through the geometric shape of the microphone array and solve it to obtain the position distance (positioning estimation) of the identification object.

[0074] Regarding the above step 1, there are many methods for estimating the delay of sound reaching different microphones. The embodiment of the present application is based on generalized cross-correlation delay estimation. In a possible implementation method, the embodiment of the present application proposes Figure 7 The schematic diagram of the delay estimation method based on generalized cross-correlation is shown in FIG. Figure 7 As shown. Step 1 includes step 1.1: calculating the cross power spectrum of the signal of the sound arriving at the two microphones. Figure 7As shown, each signal is first windowed, and after the FFT is transferred to the frequency domain, one of the signals is conjugated and then multiplied to obtain the cross-power spectrum. Step 1.2: Weighted processing of the cross-power spectrum. The weighted processing is to highlight and sharpen the correlation and weaken the irrelevant. The weighted functions include cross-correlation function, smooth coherence transform, maximum likelihood weighting, and phase transform (PHAT) weighting. Among them, PHAT phase transform weighting is the most typical weighting function. The formula is 1.0 / np.abs(f1*np.conj(f2)), f1 and f2 are the frequency domain transforms of the two signals respectively, np.abs() is a function in the NumPy library, which is used to obtain the absolute value (i.e. amplitude) of the input complex or real array, and np.conj(): is a function in the NumPy library, which is used to obtain the conjugate value of the input complex array. Step 1.3: Perform an inverse fast Fourier transform (IFFT) on the weighted power spectrum to the time domain. Step 1.4: Calculate the delay by peak detection. Specifically, find the point with the largest value in the time domain signal, i.e., the peak value. In the context of delay estimation, the peak value is used to compare the similarity of the two signals and then find the time difference (delay) between them.

[0075] The above step 2 includes: the incident angle of the far-field sound source on the microphone array is equivalent to a plane wave, and the incident angle is the same for different microphones with only a difference in distance. The incident angle of the near-field sound source on the microphone array is equivalent to a spherical wave, and the incident angle is different for different microphones with different distances. The distance r between the sound source and the microphone is greater than When d is the distance between the two microphones, λ is the wavelength of the sound wave. The distance between the microphones on the glasses is greater than 3cm, and the distance from the opposite person to the glasses microphone is definitely a far-field scene. Figure 8 is a schematic diagram of a far-field sound source position estimation method for two microphones provided in an embodiment of the present application, such as Figure 8 As shown, y1(k) and y2(k) are two microphones, s(k) is the sound source signal, and according to the trigonometric function relationship: delay ⊿T = d cos(θ) / c, c is the speed of sound 340 (m / s), d is the distance between microphones, and θ is the incident angle. From step 1 above, we can know the delay ⊿T, and then we can find the angle θ = arccos(⊿Tc / d).

[0076] Positioning on a 2D plane requires 3 microphones, and 2 microphones can only calculate the angle. The following description assumes that the glasses have 3 microphones. Fig. 9 is a schematic diagram of a near-field sound source localization estimation method using three microphones provided in an embodiment of the present application, such as Fig. 9As shown, r1 is the distance between the sound source and microphone y1(k), r2 is the distance between the sound source and microphone y2(k), and r3 is the distance between the sound source and microphone y3(k). Then the time delay between microphone 1 and microphone 2, and the time delay between microphone 1 and microphone 3, are calculated as follows:

[0077] ⊿T12=(r2-r1) / c(Equation 1)

[0078] ⊿T13=(r3-r1) / c (Equation 2)

[0079] ΔT12 and ΔT13 in the above equations 1 and 2 can be estimated based on the delay of the above generalized cross-correlation, and the angle θ can also be calculated.

[0080] from Fig. 9 The microphone geometry shown yields the following equation:

[0081] (r2) 2 =(r1) 2 +(d) 2 +2*r1*d*cos(θ1) (Equation 3)

[0082] (r3) 2 =(r1) 2 +4*(d) 2 +4*r1*d*cos(θ1) (Equation 4)

[0083] In the above equations 1 to 4, there are three unknowns r1, r2, and r3, so r1, r2, and r3 can be solved through the above four equations. Since there are three distances, in the embodiment of the present application, the average value of r1, r2, and r3 can be used as the position distance of the identification object, and then the above time difference can be calculated based on the quotient of the position distance and the sound propagation speed.

[0084] Through the above steps, the distance from the speaker to the microphone has been calculated. Since the speaker can move around at any time, the above distance calculation must also be updated regularly, so that the audio and video delay compensation is more accurate, and the model effect will be better when the voiceprint separation neural network model performs voiceprint separation.

[0085] Step S104, obtaining a first feature and a second feature of the target mixed audio data, and a third feature of the target image data, wherein the first feature is a basic attribute feature of the target mixed audio.

[0086] In some embodiments of the present application, the second feature includes at least one of the following: the direction of arrival of the sound source signal, the time of arrival of the sound source signal, and the energy of the sound source signal. The third feature includes the position of the anchor frame with the highest confidence in the target image data and the confidence of the anchor frame.

[0087] For example, the first feature mentioned above can be FBANK, MFCC. The following describes the acquisition process of the first feature using FBANK as an example. The input of this process: a segment of audio, which is collected continuously as input. The audio sampling rate is generally 16kHz, the bit width is 16bit, and it is a single channel. Output: feature value. For example, if this segment of audio is 160ms, it is divided into 16 frames. The feature of each frame is 40-dimensional data, and the feature of 16 frames is [16, 40]. Function: In order to process real-time, the audio is cut into segments, usually 10ms as a frame. Generally speaking, a segment of 10ms of audio can extract meaningful features and results. In the subsequent processing process, in order to prevent spectrum leakage, the method of partially overlapping the previous and next frames is adopted during processing. The frame length is 25ms each time, of which 15ms is historical information, that is, 15ms overlap, and only 10ms is actually moved.

[0088] BANK (Filter-Bank) is obtained by summing the squares of the power spectrum of the Mel filter and then taking the logarithm. Compared with MFCC, it omits one step of discrete cosine transform calculation. The calculation steps of FBANK are as follows Fig.10 As shown. Generally, the audio file is pre-emphasized first, in order to enhance the high-frequency signal; then it is framed and windowed. Framing is to separate the audio signal into 10ms frames, and windowing is to prevent spectrum leakage. Each time, 25ms of signal is used to calculate the feature, that is, each time it moves 10ms, and 25ms of signal is actually used, with 15ms of historical overlapping information. Then, the time domain signal is used to obtain the frequency domain signal, that is, the spectrum, through Fourier transform. The frequency domain of a certain time is accumulated to obtain the spectrum, and then the frequency is mapped to the Mel frequency scale through the Mel filter bank, and finally the logarithm is taken to obtain the FBANK feature. In the application, the Mel filter bank is generally 40 or 80, that is, each frame of audio corresponds to 40 or 80 outputs.

[0089] When the second feature includes the direction of arrival of the sound source signal, since the position and orientation of the microphone on the glasses are fixed, the direction already exists in the audio after the voices of the wearer and non-wearer are recorded. If the direction of arrival of the sound source channel in the audio can be extracted, it can assist in determining whether the current speaker is the wearer or the non-wearer. In some embodiments of the present application, a sound source localization algorithm based on a signal subspace method, such as MUSIC (Multiple Signal Classification), a beamforming method, such as SRP-PHAT (Steered Response Power-Phase Transform), and a controllable response power based on phase transform weighting is proposed. Take the MUSIC method as an example to describe the steps of direction estimation: Assume that the glasses have N microphones and M sound sources, and assume that M = 2, and find the covariance matrix of the signal; perform eigendecomposition on the covariance matrix; there are N sound source eigenvalues ​​in total, and the corresponding N eigenvectors, and obtain M sound source subspaces and NM noise subspaces. Among the N sound source eigenvalues, the M eigenvalues ​​with larger eigenvalues ​​are the eigenvalues ​​of the sound source direction (that is, the direction of arrival of the sound source signal), and the NM eigenvalues ​​with smaller eigenvalues ​​are the eigenvalues ​​of the noise direction. Search for the angles of the two sound sources with the largest current spatial spectrum. Sort the N eigenvalues ​​from large to small, traverse the first M eigenvalues ​​for each angle, and calculate the spatial spectrum. The angle with the largest spatial spectrum is the direction of the sound source. Traverse each signal source (the two largest sound sources); traverse each angle (for example, θ (0~360), with an interval of 1 degree); calculate the spatial spectrum, and the direction corresponding to the angle with the largest spatial spectrum is the direction of the sound wave. The formula for the spatial spectrum is as follows:

[0090]

[0091] Where θ is the angle and α(θ) is the steering vector, which is the ideal response of the microphone array to a sound source in a certain direction. is the characteristic vector of the noise, (·) H represents the conjugate transpose.

[0092] Since the steering vector and the noise are uncorrelated, the inner product is minimal. When the angle θ is the direction of the sound source, the steering vector and the noise are orthogonal, or in other words, they are uncorrelated and orthogonal in an ideal state. The more orthogonal, the smaller the denominator, and the denominator in an ideal state is 0. At this time, the closer the denominator is to 0, the larger the P value, and the angle corresponding to the maximum P value is the direction of the sound source. The output of this step can be the spatial spectrum of the two sound sources obtained in the above step, or it can be the spatial spectrum plus the corresponding angle value as a subsequent multimodal feature.

[0093] In some embodiments of the present application, the sound source signal energy calculation in the above second feature can be in the time domain or in the frequency domain. For example, the frequency domain energy calculation uses a short-time FFT to calculate a frequency domain signal of one frame, obtains a complex value x at N (for example, 256) frequency domain points, and then calculates the energy of the frequency domain signal. The energy calculation formula is: (x.real) 2 +x.imag 2 Using a sliding average, the sliding window length is 1 second, that is, the average of the frequency domain energy values ​​of all frames within 1 second.

[0094] The third feature of the target image data can be the target image portion of interest in an image, and the position and size of the target image portion can be obtained. Commonly used target detection neural networks include YOLO, SSD, etc. The following takes YOLO as an example to describe the target (e.g., lips) detection scheme.

[0095] YOLO is fast and can detect the location of the target (anchor box) and its category. It is a fully convolutional neural network that only contains: convolution, pooling, and fully connected layers, which is convenient for hardware acceleration. In one possible implementation, the present application embodiment provides Fig.11 A schematic diagram of a YOLO parameter list is shown, and Fig.12 Schematic diagram of another YOLO parameter list. In the embodiment of the present application, taking the first YOLO network as an example, its output includes 3 contents: category confidence, anchor box confidence, and anchor box. There are 20 categories, that is, 20 floating point confidence values, and the embodiment of the present application only cares about the lip category. Confidence values ​​of 2 anchor boxes, 2 floating point numbers. Position values ​​of 2 anchor boxes, 4 coordinate values ​​for each anchor box. So the output of the first YOLO network is 30 values. In the embodiment of the present application, since only the lip part is of interest, it is sufficient to detect the lip position in the picture. Since there may be multiple people in front of the wearer of glasses, only the lips of the closest person are of interest, that is, there may be only 1 or 0 detected targets. If there are multiple, the one with the largest size is selected, which is equivalent to the closest one. If the person is standing far away, that is, the lip size is relatively small, less than a certain threshold value (for example, less than 1 / 20 of the picture), it is not considered that the lips are detected, because this should not be a normal translation scene. The output of this step is the image segmented at the position of the anchor box with the highest confidence, as well as the confidence of the anchor box.

[0096] Step S106, inputting the first feature into a voiceprint separation neural network model, and fusing the second feature and the third feature in the voiceprint separation neural network model to separate the voice of the identified object from the target mixed audio data.

[0097] In some embodiments of the present application, there is provided Fig.13The schematic diagram of the voiceprint separation neural network model structure is shown in Fig.13 As shown, it includes an encoding network and a decoding network, and the encoding network and the decoding network include multiple network blocks. Optionally, the decoding network also includes a fusion module, which is used to fuse the above second feature and the above third feature. In a possible implementation, the embodiment of the present application also provides Fig.14 The schematic diagram of the network block structure shown in the figure includes CNN, RNN / LSTM, and CNN. There are many ways to fuse the fusion module. For example, features A and B are one-dimensional vectors, and they can be fused together using the following fusion methods:

[0098] Dimension expansion fusion: [A, B]

[0099] Plus fusion: A+B, the premise is that the dimensions of A and B are consistent

[0100] Multiplication Fusion: A*B

[0101] In the embodiment of the present application, dimension expansion fusion is used. Since the dimension expansion fusion method is used, it indicates that the parameters will increase. If the fusion is in the front, the dimension of the subsequent network will increase. Therefore, the fusion module is placed at the end of the entire neural network as much as possible, such as placing the fusion module in the last network block of the decoding network. Fig.15 Another schematic diagram of the structure of a voiceprint separation neural network model is shown, in which only parameters need to be added to the last submodule.

[0102] Since multimodal features need to have temporal contextual information, for example, a single lip image cannot identify whether someone is speaking, but multiple images before and after in time are equivalent to the effect of a video, so it is necessary to retain the information of the previous period of time of these features. This information can be saved using the RNN / LSTM time recurrent neural network. In other words, multimodal features need to be fused before the RNN / LSTM. Therefore, in some embodiments of the present application, the architecture of the voiceprint separation neural network model can be that the above-mentioned network block includes multiple network layers, the first layer of the multiple network layers is a convolutional neural network CNN, the second layer is a recurrent neural network RNN ​​or a long short-term memory neural network LSTM, and the third layer is CNN. The fusion module is located before the recurrent neural network RNN ​​or the long short-term memory neural network LSTM, such as Fig.16 Another schematic diagram of the structure of a voiceprint separation neural network model is shown.

[0103] Alternatively, in a possible implementation, the present application embodiment provides the following Fig.17 The network block structure diagram shown in Fig.17As shown, the first layer of the multiple network layers is a convolutional neural network CNN, the second layer is a merging processing module with a memory unit, and the third layer is a CNN. The memory unit is used to store historical features, and the merging processing module is used to merge the historical features with the current features and update the current features to the memory unit. The fusion module is located before the merging processing module, such as Fig.18 Another structural diagram of a voiceprint separation neural network model is shown. The memory unit acts as a memory unit to expand the time dimension. The feature data stored in the memory unit is similar to a 2-dimensional or multi-dimensional floating point matrix.

[0104] After separating the sound of the recognized object from the target mixed audio data, speech recognition can be done on the mobile phone or on the cloud server. Usually, the recognition effect on the cloud server is better, but the delay is also greater because of the network transmission delay, especially when there is no wireless network or the network signal is poor. In the translation scenario, the translation is usually implemented in the cloud. Then, the translation result is transmitted to the glasses. Alternatively, the mobile phone obtains the translation result text and transmits it to the glasses via Bluetooth. Finally, the translation result is displayed on the glasses screen.

[0105] Since there are short pauses in speech, after separating the speech of the identification object (non-wearer) in the embodiment of the present application, a "human voice state" can also be set. If there is a subsequent frame of silent data, it is not considered that the state changes from "human voice state" to "silent state". After a certain period of time, for example, 300ms of continuous silence, the non-wearer will be considered to have entered the "silent state". As shown in Table 2.

[0106] Table 2

[0107] Double-talk judgment result state No one is talking Mute state Only the wearer speaks Mute state Only non-wearers are talking Vocal Status Wearers and non-wearers alike are talking Vocal Status

[0108] In summary, in view of the different acquisition delays of sound and video, and the relatively large sound delay, the embodiment of the present application performs time synchronization processing on the target mixed audio data collected by the microphone on the smart glasses and the target image data of the identified object collected by the camera. In addition, when performing voiceprint separation through the voiceprint separation neural network model, in addition to using the characteristics of the target mixed audio data itself (the first feature), the second feature of the target mixed audio data and the third feature of the target image data are also integrated, which improves the accuracy of voiceprint separation, thereby solving the technical problem of low accuracy of the voiceprint separation method in the related art.

[0109] In addition, the related art uses front fusion or mid fusion, while the embodiment of the present application adopts post fusion, specifically, fusion is performed at the last network block of the decoding network, or before the memory processing unit of the decoding network, to reduce parameters and calculation amount.

[0110] In addition, since it is an audio timing feature, the lip movements are also time-related. In order to extract this temporal feature, most of the designed networks use recurrent neural networks such as RNN / LSTM / GRU, which brings about the problem of increased calculation amount and parameters, which is not suitable for low-power devices. The embodiment of the present application uses a merging processing module with a memory unit instead of a recurrent neural network, which is more flexible, has fewer parameters, and has a smaller amount of calculation.

[0111] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a voiceprint separation device for smart glasses. Fig.19 The figure shows a structural block diagram of a voiceprint separation device for smart glasses according to an embodiment of the present application, including:

[0112] A synchronization module 1902 is used to perform time synchronization processing on the target mixed audio data collected by the microphone on the smart glasses and the target image data of the recognized object collected by the camera;

[0113] An acquisition module 1904 is used to acquire a first feature and a second feature of the target mixed audio data, and a third feature of the target image data, wherein the first feature is a basic attribute feature of the target mixed audio;

[0114] The separation module 1906 is used to input the first feature into the voiceprint separation neural network model, and fuse the second feature and the third feature in the voiceprint separation neural network model to separate the sound of the identified object from the target mixed audio data.

[0115] In some embodiments of the present application, the second feature includes at least one of the following: the direction of arrival of the sound source signal, the time of arrival of the sound source signal, and the energy of the sound source signal. The third feature includes the position of the anchor frame with the highest confidence in the target image data and the confidence of the anchor frame.

[0116] The above-mentioned voiceprint separation neural network model includes an encoding network and a decoding network. The encoding network and the decoding network include multiple network blocks. The decoding network also includes a fusion module, and the fusion module is used to fuse the second feature and the third feature.

[0117] In some embodiments of the present application, the fusion module is located in the last network block of the decoding network.

[0118] The above-mentioned network block includes multiple network layers, the first layer of the multiple network layers is a convolutional neural network CNN, the second layer is a memory processing unit, and the third layer is CNN. The memory processing unit includes a recurrent neural network RNN ​​or a long short-term memory neural network LSTM or a merging processing module with a memory unit. The memory unit is used to store historical features, and the merging processing module is used to merge the historical features with the current features, and update the current features to the memory unit.

[0119] In some embodiments of the present application, the fusion module is located before the memory processing unit.

[0120] The synchronization module 1902 is also used to obtain the time difference between the audio transmission and the light transmission from the identified object; the time difference is used to correct the acquisition time of the target mixed audio data forward. Optionally, the time difference is obtained based on the quotient of the estimated distance corresponding to the current application scenario and the sound propagation speed. The smart glasses include a microphone array composed of multiple microphones, and obtain the time difference between the audio transmission and the light transmission from the identified object, including: determining the time when the target mixed audio data arrives at each microphone respectively; according to the time when the target mixed audio data arrives at each microphone respectively, and the array shape of the microphone array, calculate the position distance of the identified object; according to the quotient of the position distance and the sound propagation speed, calculate the time difference.

[0121] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0122] The embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.

[0123] An embodiment of the present application also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0124] The embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0125] refer to Fig. 20, the structural block diagram of the electronic device that can be used as the server or client of the embodiment of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0126] like Fig. 20 As shown, the electronic device includes a computing unit 2001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2002 or a computer program loaded from a storage unit 2008 into a random access memory (RAM) 2003. In the RAM 2003, various programs and data required for the operation of the electronic device can also be stored. The computing unit 2001, the ROM 2002, and the RAM 2003 are connected to each other via a bus 20020. An input / output (I / O) interface 2005 is also connected to the bus 20020.

[0127] Multiple components in the electronic device are connected to the I / O interface 2005, including: an input unit 2006, an output unit 2007, a storage unit 2008, and a communication unit 2009. The input unit 2006 can be any type of device that can input information to the electronic device, and the input unit 2006 can receive input digital or character information, and generate key signal input related to the user settings and / or function control of the electronic device. The output unit 2007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 2008 can include but is not limited to a disk, an optical disk. The communication unit 2009 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0128] The computing unit 2001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 2001 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 2001 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present application may be implemented as a computer program, which is tangibly included in a machine-readable medium, such as a storage unit 2008. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 2002 and / or a communication unit 2009. In some embodiments, the computing unit 2001 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0129] The computer program for implementing the method of the embodiment of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable signal medium can include but is not limited to electronic, magnetic, optical, electromagnetic or infrared systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0131] It should be noted that the term "including" and its variations used in the embodiments of the present application are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0132] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0133] The various steps described in the method implementation methods provided in the embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present application is not limited in this respect.

[0134] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0135] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. Voiceprint separation method for smart glasses, including: Performing time synchronization processing on the target mixed audio data collected by the microphone on the smart glasses and the target image data of the identified object collected by the camera; Acquire a first feature and a second feature of the target mixed audio data, and a third feature of the target image data, wherein the first feature is a basic attribute feature of the target mixed audio; The first feature is input into a voiceprint separation neural network model, and the second feature and the third feature are fused in the voiceprint separation neural network model to separate the voice of the identified object from the target mixed audio data.

2. The method according to claim 1, wherein: The second feature includes at least one of the following: the direction of arrival of the sound source signal, the time of arrival of the sound source signal, and the energy of the sound source signal; and / or The third feature includes the position of the anchor box with the highest confidence in the target image data and the confidence of the anchor box.

3. The method according to claim 1, wherein: The voiceprint separation neural network model includes an encoding network and a decoding network. The encoding network and the decoding network include multiple network blocks. The decoding network also includes a fusion module. The fusion module is used to fuse the second feature and the third feature. The fusion module is located in the last network block of the decoding network.

4. The method according to claim 3, wherein: The network block includes multiple network layers, the first layer of the multiple network layers is a convolutional neural network CNN, the second layer is a memory processing unit, and the third layer is CNN. The memory processing unit includes a recurrent neural network RNN ​​or a long short-term memory neural network LSTM or a merging processing module with a memory unit. The memory unit is used to store historical features, and the merging processing module is used to merge the historical features with the current features, and update the current features to the memory unit.

5. The method according to claim 4, wherein: The fusion module is located before the memory processing unit.

6. The method according to claim 1, wherein: The time synchronization processing of the target mixed audio data collected by the microphone on the smart glasses and the target image data of the recognized object collected by the camera includes: A time difference between audio transmission and light transmission from the identified object is obtained, and the time difference is used to correct the acquisition time of the target mixed audio data forward.

7. The method according to claim 6, wherein: The time difference is obtained based on the quotient of the estimated distance corresponding to the current application scenario and the sound propagation speed.

8. The method according to claim 6, wherein: The smart glasses include a microphone array consisting of a plurality of microphones, and obtaining a time difference between audio transmission and light transmission from the identified object includes: Determine the time when the target mixed audio data arrives at each microphone respectively; Calculate the position distance of the identified object according to the time when the target mixed audio data arrives at each microphone and the array shape of the microphone array; The time difference is calculated based on the quotient of the position distance and the sound propagation speed.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium having a computer program stored therein, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

Images