Attitude detection method, device, electronic device and computer readable storage medium
By collecting the attitude energy information and feature vectors of the voice signal and determining the weight value in combination with the neural network, the problem of inability to effectively detect user attitudes in the prior art is solved, and efficient noise suppression and improved voice quality in multi-microphone devices are achieved.
Patent Information
- Application Number
- CN202010725583.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-02-20
- Filing Date
- 2020-07-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-07-24
AI Technical Summary
The prior art cannot effectively detect the attitude of a user when the handheld terminal device, resulting in a degradation of the performance of the noise suppression function in an abnormal attitude, and the inability to maximize the use of the advantages of multi-microphones.
The voice signal is collected through at least two voice acquisition devices, the attitude energy information of the voice signal is obtained, the projection energy calculation is performed using the predetermined attitude and feature vectors, and the weight value is determined in combination with the neural network to realize the weighting processing of the voice signal, thereby determining the user's attitude And optimize noise suppression processing.
Maintain robustness in non-stationary noise and low signal-to-noise ratio, supports any two or more microphone devices, reduces algorithm complexity, improves posture detection accuracy and call voice quality.
Smart Images

Figure CN113284504B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a posture detection method, device, electronic device and computer-readable storage medium. Background Art
[0002] With the development of the terminal equipment industry, users have higher and higher requirements for the call quality of terminal equipment. Existing terminal equipment generally has two or more microphones (Microphone, mic). Thanks to the multi-microphone noise reduction method, multi-microphone terminal equipment can obtain better call voice quality than single-microphone terminal equipment. Whether the multi-microphone noise reduction method is effective depends largely on the posture of the user holding the terminal device when using the terminal device for voice calls. However, the posture of the handheld terminal device is different for different people or the same person at different times. Therefore, detecting the posture of the handheld terminal device when the user uses the terminal device for voice calls plays a vital role in the noise suppression function.
[0003] At present, some posture detection methods based on sensors (such as cameras, gyroscopes, gravity sensors, etc.) cannot obtain the posture of the terminal device relative to the face, so it is necessary to improve the existing posture detection methods. Summary of the invention
[0004] The purpose of this application is to solve at least one of the above technical defects. The technical solutions provided by the embodiments of this application are as follows:
[0005] In a first aspect, an embodiment of the present application provides a posture detection method, including:
[0006] Collecting voice signals during a voice call by at least two voice collection devices;
[0007] Obtaining gesture energy information corresponding to the speech signal;
[0008] According to the posture energy information, the posture of the user during the voice call is determined from various predetermined postures.
[0009] In an optional embodiment of the present application, obtaining gesture energy information corresponding to the speech signal includes:
[0010] The projection energy of the speech signal corresponding to each predetermined posture is obtained.
[0011] In an optional embodiment of the present application, obtaining the projection energy of the speech signal corresponding to each predetermined posture includes:
[0012] Obtaining the projection energy of each frequency point of the speech signal corresponding to each predetermined posture;
[0013] Get the weight value corresponding to each frequency point;
[0014] The projection energy of the speech signal corresponding to each predetermined posture is determined according to the weight values corresponding to each frequency point and the projection energy of each frequency point corresponding to each predetermined posture.
[0015] In an optional embodiment of the present application, obtaining the projection energy of each frequency point of the speech signal corresponding to each predetermined posture includes:
[0016] Obtain the eigenvector corresponding to the speech signal, where the eigenvector contains the eigenvalues corresponding to each frequency point;
[0017] Normalizing the feature vector to obtain a normalized feature vector corresponding to the speech signal;
[0018] The projection energy of each frequency point corresponding to each predetermined posture is determined according to the normalized eigenvector and the eigenmatrices corresponding to each predetermined posture.
[0019] In an optional embodiment of the present application, obtaining a feature vector corresponding to a speech signal includes:
[0020] Acquire at least two frequency domain signals corresponding to the speech signal;
[0021] For each frequency point, the eigenvalues of each frequency domain signal at that frequency point are combined to obtain the eigenvector of the speech signal.
[0022] In an optional embodiment of the present application, before normalizing the feature vector, the method further includes:
[0023] According to the predetermined compensation parameters, the frequency response compensation processing is performed on the eigenvector to obtain the amplitude-corrected eigenvector.
[0024] In an optional embodiment of the present application, for at least one predetermined posture, the corresponding feature matrix is obtained in the following manner:
[0025] Determine the distance between the sample sound source and each voice collection device respectively;
[0026] Determine a feature vector corresponding to a predetermined posture according to the distance between the sample sound source and each voice collection device;
[0027] Based on the feature vector corresponding to the predetermined posture, a feature matrix corresponding to the predetermined posture is determined.
[0028] In an optional embodiment of the present application, obtaining weight values corresponding to each frequency point includes:
[0029] Get the predetermined weight corresponding to each frequency point.
[0030] In an optional embodiment of the present application, obtaining weight values corresponding to each frequency point includes:
[0031] According to the projection energy of each frequency point corresponding to each predetermined posture and / or the energy information of the voice signal, a neural network is determined through weights to determine the weights corresponding to each frequency point.
[0032] In an optional embodiment of the present application, the weight determination neural network includes a control subnetwork and a calculation subnetwork;
[0033] According to the projection energy of each frequency point corresponding to each predetermined posture and / or the energy information of the voice signal, a neural network is determined through weights to determine the weights corresponding to each frequency point, including:
[0034] Determine, by the control subnetwork, a signal-to-noise ratio characterization value of the speech signal according to energy information of the speech signal, and determine, according to the signal-to-noise ratio characterization value, whether the weight value of each frequency point is a predetermined weight;
[0035] If not, the weight value of each frequency point is determined by the calculation sub-network according to the projection energy of each frequency point corresponding to each predetermined posture.
[0036] In an optional embodiment of the present application, the control subnetwork extracts features of energy information of the speech signal via a plurality of cascaded first feature extraction layers, and obtains a signal-to-noise ratio representation value according to the extracted features via a classification layer;
[0037] The computing subnetwork extracts features of the projection energy of each frequency point corresponding to each predetermined posture through a plurality of cascaded second feature extraction layers, and obtains the weight of each frequency point according to the extracted features through a linear regression layer.
[0038] In an optional embodiment of the present application, the second feature extraction layer merges the extracted features with the features output by the corresponding first feature extraction layer in the control subnetwork, and outputs the merged features.
[0039] In an optional embodiment of the present application, determining the projection energy of the speech signal corresponding to each predetermined posture according to the weight values corresponding to each frequency point and the projection energy of each frequency point corresponding to each predetermined posture includes:
[0040] For each predetermined posture, the projection energy of each frequency point corresponding to the predetermined posture is weighted according to the weight value corresponding to each frequency point;
[0041] For each predetermined posture, the weighted projection energies of each frequency point corresponding to the predetermined posture are accumulated to obtain the projection energy of the speech signal corresponding to the predetermined posture.
[0042] In an optional embodiment of the present application, determining the posture of the user during the voice call from various predetermined postures according to the posture energy information includes:
[0043] The predetermined posture corresponding to the maximum posture energy information is selected to obtain the posture information of the user during the voice call.
[0044] In an optional embodiment of the present application, it also includes:
[0045] Based on the user's posture information during the voice call, the voice signal is denoised to obtain a noise-reduced voice signal.
[0046] In a second aspect, an embodiment of the present application provides a posture detection device, including:
[0047] A voice collection module, used to collect voice signals during a voice call through at least two voice collection devices;
[0048] An energy acquisition module, used to obtain gesture energy information corresponding to the voice signal;
[0049] The posture determination module is used to determine the posture of the user during the voice call from various predetermined postures according to the posture energy information.
[0050] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor;
[0051] A computer program is stored in the memory;
[0052] A processor is used to execute a computer program to implement the method provided in the embodiment of the first aspect or any optional embodiment of the first aspect.
[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method provided in the embodiment of the first aspect or any optional embodiment of the first aspect is implemented.
[0054] The beneficial effects of the technical solution provided by this application are:
[0055] By pre-setting multiple predetermined postures, and then determining the user's posture during the call from the pre-set multiple predetermined postures based on the posture energy information of the voice signals collected by multiple voice collection devices during the voice call, the user's posture during the call is obtained, that is, the user posture detection result is obtained, which can provide a reference for enhancing the call voice quality and optimize the effect of noise suppression processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.
[0057] Figure 1A schematic diagram of a terminal device including two microphones in an example of an embodiment of the present application;
[0058] Figure 2a A schematic diagram of a user's posture in a normal handheld mode in an embodiment of the present application;
[0059] Figure 2b This is a waveform diagram of a voice signal collected by a microphone in a normal handheld mode in an embodiment of the present application;
[0060] Figure 3a A schematic diagram of a user's posture in an abnormal hand-held mode in an embodiment of the present application;
[0061] Figure 3b This is a waveform diagram of a voice signal collected by a microphone in an abnormal handheld mode in an embodiment of the present application;
[0062] Figure 4 A flowchart of a posture detection method provided in an embodiment of the present application;
[0063] Figure 5a A schematic diagram of a predetermined posture 1 in an example of an embodiment of the present application;
[0064] Figure 5b A schematic diagram of a predetermined posture 2 in an example of an embodiment of the present application;
[0065] Figure 5c A schematic diagram of a predetermined posture 3 in an example of an embodiment of the present application;
[0066] Figure 5d A schematic diagram of a predetermined posture 4 in an example of an embodiment of the present application;
[0067] Figure 5e A schematic diagram of the distance between a sound source and a microphone in each predetermined posture and user posture in an example of an embodiment of the present application;
[0068] Figure 5f A schematic diagram of feature vectors corresponding to each predetermined posture and user posture in an example of an embodiment of the present application;
[0069] Figure 5g A schematic diagram of normalized feature vectors corresponding to each predetermined posture and user posture in an example of an embodiment of the present application;
[0070] Figure 6a It is a schematic diagram of a user posture and normalized feature vectors of each predetermined posture in the same coordinate system in an example of an embodiment of the present application;
[0071] Figure 6b for Figure 6aThe example shown is a schematic diagram of energy projection of the normalized feature vector of the user's posture on the feature vectors of each predetermined posture;
[0072] Figure 7 A structural diagram of a call voice enhancement system provided in an embodiment of the present application;
[0073] Figure 8 A schematic diagram of a processing flow in a projection energy calculation module in an example of an embodiment of the present application;
[0074] Fig. 9 A schematic diagram of a feature matrix calculation process for a predetermined posture provided in an embodiment of the present application;
[0075] Fig.10 A schematic diagram of a posture detection process when the weights of each frequency point are predetermined weights in an example of an embodiment of the present application;
[0076] Fig.11 This is a schematic diagram of a posture detection process when the weights of each frequency point are determined by a neural network in an example of an embodiment of the present application;
[0077] Fig.12 A schematic diagram of projection energy accumulation in an example of an embodiment of the present application;
[0078] Fig.13 A schematic diagram of a weight determination neural network structure in an example of an embodiment of the present application;
[0079] Fig.14 A schematic diagram of adjusting the initial weight value in an example of an embodiment of the present application;
[0080] Fig.15 A schematic diagram of the energy corresponding to two frequency points in an example of an embodiment of the present application;
[0081] Fig.16a A schematic diagram of the projection energy of frequency point 2 at each predetermined posture in an example of an embodiment of the present application;
[0082] Fig.16b A schematic diagram of the projection energy of frequency point 3 at each predetermined posture in an example of an embodiment of the present application;
[0083] Fig.17 A schematic diagram of a terminal device including three microphones in an example of an embodiment of the present application;
[0084] Fig.18 A flowchart of a posture detection method for a situation involving three or more voice collection devices provided in an embodiment of the present application;
[0085] Fig.19A waveform diagram of a speech signal collected by each microphone and a schematic diagram of a gesture detection structure in an example of an embodiment of the present application;
[0086] Fig. 20 A schematic diagram of a voice call process in an example of an embodiment of the present application;
[0087] Fig.21 A schematic diagram of a flow chart of another posture detection method provided in an embodiment of the present application;
[0088] Fig. 22 A structural block diagram of a posture detection device provided in an embodiment of the present application;
[0089] Fig.23 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0090] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.
[0091] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0092] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0093] Manufacturers of terminal devices (such as mobile phones, PADs, smart watches and other devices) will obtain better call voice quality by setting two or more microphones on the terminal devices. Specifically, multiple voice signals during a voice call are collected through multiple microphones respectively, and then noise suppression processing is performed based on the multiple voice signals to obtain a voice signal with a higher signal-to-noise ratio. The above process of performing noise suppression processing on the voice signal during a voice call can be called call voice enhancement.
[0094] Take dual-microphone terminal equipment as an example. Figure 1 As shown, the two microphones are located at the upper and lower ends of the terminal device. When the user is talking in handset mode, it can be considered that the voice signal collected by the lower microphone contains the user's voice and noise, while the voice signal collected by the upper microphone contains only noise. Then, the voice signal collected by the upper microphone is used as a reference to eliminate the noise in the voice signal collected by the lower microphone to obtain clean voice.
[0095] In general handheld mode, the user's posture (handheld device posture) and the voice signals collected by the upper and lower microphones are as follows: Figure 2a-2b As shown in the figure, at this time, the microphone at the bottom of the terminal device is close to the user's mouth, while the microphone at the top is far from the mouth. The voice signal collected by the bottom microphone contains more user voice components than the voice signal collected by the bottom microphone, while the signal of the top microphone is mostly composed of noise. In this way, the voice signal collected by the top microphone is used as a reference for the noise signal to eliminate the noise in the voice signal collected by the bottom microphone, and finally a clean voice is obtained. This method can take advantage of multiple microphones to obtain a good noise reduction effect, while introducing very little voice distortion. This noise suppression processing method can be called "multi-microphone noise reduction method".
[0096] However, in some abnormal handheld modes, for example, the user Figure 3a When making a call in the abnormal posture shown in the figure, the voice signals collected by the upper and lower microphones are as follows Figure 3b As shown, at this time, the microphone at the bottom of the terminal device is far away from the user's mouth and no longer has a distance advantage compared to the upper microphone. The voice components contained in the voice signals collected by the upper and lower microphones are basically the same. If the voice signal collected by the upper microphone is used as a noise signal reference to eliminate the noise in the signal collected by the lower microphone, it will cause serious damage to the voice signal. Therefore, before performing noise suppression processing, the user's posture needs to be detected. When it is detected that the user uses an abnormal posture for a voice call, the voice signal collected by the upper microphone is no longer used as a noise signal reference, and only the voice signal collected by the lower microphone is used for single-channel noise reduction. This noise suppression processing method can be called "single microphone noise reduction method."
[0097] From the above, we can see that: Figure 2a In the normal posture shown, speech enhancement processing can be performed by multi-microphone noise reduction. Figure 3a In the abnormal posture shown, speech enhancement processing can be performed by a single microphone noise reduction method. Therefore, before performing speech enhancement processing on the speech signal, it is necessary to determine the user's posture.
[0098] Currently, there is no effective method for detecting the user's posture in handheld mode during a voice call. In the existing call voice enhancement solutions, there is no independent module responsible for detecting the posture of the user's handheld device during a voice call. Usually, the parameters in the noise reduction module are adjusted so that the performance of the noise reduction module is within an acceptable range under abnormal postures. At the same time, some posture detection methods based on the terminal device's own sensors (such as cameras, gyroscopes, gravity sensors, etc.) can obtain the absolute posture of the terminal device relative to the horizontal plane, but cannot obtain the posture of the terminal device relative to the face, so they are not suitable for terminal device posture detection for handheld calls, that is, they are not suitable for detecting the user's posture before performing noise suppression processing.
[0099] The existing speech enhancement methods (used to detect the user's current holding posture of the mobile phone) mainly include two categories, one of which is based on the method of comparing signal spectrum components for posture detection, and the other is based on the method of cross spectrum power (CSP) for posture detection.
[0100] Specifically, the method based on comparing the signal spectrum components is to focus on whether the information collected from the two signals has sufficient "distinction", that is, to determine the energy, spectrum similarity or the probability of speech existence calculated independently from each microphone signal based on the information collected from the two signals, so as to judge whether the two signals have sufficient distinction, thereby judging the user's current holding posture of the mobile phone;
[0101] Furthermore, the CSP-based method extracts the phase difference between the two signals in the spectrum domain, converts the phase difference to the time domain, obtains the relative delay of the main components of the signal, and determines the speaker's posture based on the delay.
[0102] However, the method based on spectrum component comparison is obviously helpless for signals with low signal-to-noise ratio (SIGNAL NOISE RATIO, SNR), that is, it will fail in high ambient noise. Moreover, when the ambient sound is human voice noise or other non-stationary noise, the probability of speech existence calculated independently based on each microphone signal will no longer be accurate, which will also cause the algorithm to fail. The failure of the detection algorithm will lead to the complete inability to determine the speaker's posture; the method based on CSP has a certain tolerance for low signal-to-noise ratio and a certain robustness to non-stationary noise (it will still fail in the case of low signal-to-noise ratio and non-stationary noise), but due to the inherent limitations of the algorithm, it can only support two signals to participate in the operation, and cannot maximize the ability of mobile phones with more than two microphones. In addition, the CSP algorithm requires the entire spectrum of the two signals to participate in the operation (that is, if the signal is 8000hz, then the signals from 0 to 8000hz need to participate in the operation), and includes a large number of division operations and additional inverse fast Fourier transform (Fast Fourier Transform, FFT) operations. Its complexity is relatively high and the burden on the hardware is heavy, as shown in Table 1.
[0103] Table 1
[0104]
[0105] The posture detection method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application are intended to solve the above technical problems in the prior art.
[0106] The gesture detection method provided in the embodiment of the present application has strong robustness to non-stationary noise and low SNR conditions, and the embodiment of the present application can support devices with more than two microphones, and the number of microphones corresponds to the dimension of the feature vector, that is, when a new microphone is added to the device, it only corresponds to one more dimension of the feature vector in the embodiment of the present application, without changing the basic architecture of the method. At the same time, because the predetermined gesture is introduced, the complexity of the algorithm is greatly reduced. At the same time, in the embodiment of the present application, the weight is used to determine the neural network to select and utilize the spectrum of the microphone signal (the weight is used to determine the neural network to weight different frequencies, and the frequencies with high signal-to-noise ratios are weighted higher, and the frequencies with low signal-to-noise ratios are weighted lower), further enhancing the robustness and accuracy of gesture detection, as shown in Table 2.
[0107] Table 2
[0108]
[0109] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0110] Figure 4 A flow chart of a posture detection method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the method may include:
[0111] Step S401: collecting voice signals during a voice call by using at least two voice collection devices.
[0112] Each voice signal includes the voice of the user (ie, the speaker) during the voice call and may also include noise during the voice call.
[0113] Step S402, obtaining gesture energy information corresponding to the speech signal.
[0114] The posture energy information corresponding to the speech signal is obtained, that is, the projection energy of the speech signal corresponding to each predetermined posture is obtained.
[0115] Step S403: determining the posture of the user during the voice call from various predetermined postures according to the posture energy information.
[0116] The user's posture may be understood as the positional relationship between a communication device (also referred to as a terminal device) used by the user and the user's face.
[0117] Specifically, for the embodiment of the present application, when the user is making a voice call through the terminal device, the voice signal input by the user during the voice call is obtained at preset intervals, and the posture detection is performed on the voice signal input by the user (that is, the posture of the user currently holding the terminal device is detected). For example, when the user is making a call through the terminal device, the voice signal of the user is obtained every 20 seconds, and the posture detection is performed on the voice signal of the user;
[0118] Alternatively, in an embodiment of the present application, when a user is making a voice call through a terminal device, when a gesture detection instruction triggered by the user is detected, a voice signal input by the user is obtained, and gesture detection is performed on the voice signal input by the user; for example, when a user is making a voice call through a terminal device, the user can trigger the terminal device to display a gesture detection instruction to trigger gesture detection of the voice signal input by the user;
[0119] Furthermore, in the embodiment of the present application, the above two methods of triggering gesture detection can be combined, that is, when the user is making a voice call through the terminal device and a gesture detection instruction triggered by the user is detected, the user's voice signal is obtained at a preset time interval, and gesture detection is performed on the user's voice signal. In the embodiment of the present application, any method of triggering the acquisition of the user's voice signal and performing gesture detection based on the user's voice signal is within the protection scope of the embodiment of the present application.
[0120] Specifically, a plurality of predetermined postures can be pre-set, and the feature vectors corresponding to each predetermined posture can be obtained, and then the feature vector corresponding to the user posture (the feature vector of the voice signal) can be obtained, and the feature vector corresponding to the user posture can be compared with the feature vectors corresponding to each predetermined posture, and the predetermined posture whose feature vector is closest to the feature vector corresponding to the user posture in each predetermined posture is determined as the user's posture, that is, the posture detection result is obtained. In the embodiment of the present application, a normalized feature vector is used, so that the feature vector corresponding to the user posture is "closest" to the feature vector corresponding to a predetermined posture, that is, the projection energy of the feature vector corresponding to the user posture under the feature vector corresponding to the predetermined posture is the largest.
[0121] The solution provided in the embodiment of the present application pre-sets multiple predetermined postures, and then determines the user's posture during the call from the pre-set multiple predetermined postures based on the posture energy information of the voice signals collected by multiple voice collection devices during the voice call, that is, obtains the user posture detection result, which can provide a reference for enhancing the call voice quality and optimize the effect of noise suppression processing.
[0122] For example, assume that four predetermined postures are preset, namely posture 1, posture 2, posture 3 and posture 4. Figure 5a-5d As shown, in posture 1, the angle between the central axis of the call device (such as a terminal device) and the central axis of the user's face is -30°, the angle between the central axis of the call device and the central axis of the user's face in posture 2 is 0°, the angle between the central axis of the call device and the central axis of the user's face in posture 3 is 30°, and the angle between the central axis of the call device and the central axis of the user's face in posture 4 is 60°. The user posture can be determined by the relative position of the sound source (user's mouth) and the microphone of the terminal device. In a terminal device including two voice collection devices (an upper microphone and a lower microphone, respectively), the user posture can be determined based on the distance from the sound source to the upper microphone and the distance from the sound source to the lower microphone. Under each predetermined posture and user posture, the distance from the sound source to the upper and lower microphones is as shown in FIG. Figure 5eThen, a rectangular coordinate system is constructed with the longitudinal center axis of the terminal device as the Y axis and the transverse center axis of the terminal device as the X axis. The feature vector corresponding to each posture is determined in the rectangular coordinate system. The starting point of the feature vector is the origin of the rectangular coordinate system, the direction points to the sound source, and the length of the feature vector indicates the size of the corresponding speech energy. The feature vectors corresponding to each predetermined posture and user posture are shown in Figure 5f As shown. Figure 5f It can be seen from , that the feature vector corresponding to the user posture is closest to the feature vector corresponding to posture 2. Figure 5g As shown in Figure 2, in order to further eliminate the influence of speech energy, a normalization step is used to calculate the Figure 5f Normalize the eigenvectors in . Figure 5g It can also be seen that the feature vector corresponding to the user posture is closest to the feature vector corresponding to posture 2.
[0123] Furthermore, the normalized feature vectors corresponding to the four predetermined postures and the normalized feature vector corresponding to the user posture can be placed in the same coordinate system, such as Figure 6a As shown, the normalized feature vectors of each predetermined posture are represented by dotted arrows, and the normalized feature vectors corresponding to the user posture are represented by solid arrows. It can be seen that the arrow corresponding to posture 2 in the figure is closest to the arrow corresponding to the user posture. Therefore, it is considered that the normalized feature vector corresponding to the user posture is closest to the normalized feature vector corresponding to posture 2 (it can also be understood as the most similar). In order to accurately measure the similarity between the normalized feature vector corresponding to the user posture and the normalized feature vectors corresponding to each predetermined posture, it is necessary to perform energy projection of the normalized feature vector corresponding to the user posture on the normalized feature vector corresponding to each predetermined posture, and the greater the projection energy, the higher the corresponding similarity. Figure 6b As shown, the normalized feature vector of each predetermined posture is represented by a dotted arrow, and the normalized feature vector corresponding to the user posture is represented by a solid arrow. Each dotted line is a corresponding energy projection operation. The dotted line segments corresponding to brackets 1, 2, 3, and 4 are the projection energies on each predetermined posture, where the longer the projection energy, the higher the similarity. Figure 6b It can be seen that the projection energy corresponding to bracket 2 is the longest, that is, the normalized feature vector corresponding to the user posture has the largest projection energy on posture 2, and the corresponding similarity is the highest. Then posture 2 is determined as the user posture, and the posture detection result is obtained.
[0124] For the embodiments of the present application, after determining the posture information of the user during the voice call in the above manner, the collected voice signal can be voice enhanced by using a voice enhancement processing method that matches the determined posture information, and the enhanced voice signal can be sent to the other end of the voice call.
[0125] When the gesture detection result is used for call voice quality enhancement, the structure of the call voice quality enhancement system is as follows: Figure 7 As shown, the system may include: a projection energy calculation module 701, a posture matching detection module 702 and a noise suppression module 703. Two microphones are provided on the call device 704, namely, an upper microphone and a lower microphone. During the voice call, the voice signal collected by the upper microphone is recorded as s1, and the voice signal collected by the lower microphone is recorded as s2. The projection energy calculation module 701 calculates the projection energy e of each frequency point in the voice signal s1 and the voice signal s2 at each predetermined posture. (j,k) , wherein the posture matching detection module 702 obtains the posture detection result h based on the projection energy corresponding to each frequency index , and then input the posture detection result into the noise suppression module 703 as a reference for noise suppression processing, and finally output the noise-reduced speech signal y. The noise suppression module 703 uses this posture detection result to avoid speech distortion caused by over-estimation of noise, thereby optimizing the effect of noise suppression processing and improving the quality of call voice.
[0126] Reference again Figure 7 , the above-mentioned projection energy calculation module and posture matching detection module can constitute a posture detection module, wherein the processing of the projection energy calculation module is divided into two parts: offline processing and online processing. The offline processing part may include: presetting multiple predetermined postures and obtaining the normalized feature vector of each predetermined posture; the online processing part may include: performing short-time Fourier transform on the voice signal collected from the microphone, merging each voice signal according to the frequency point, and obtaining the normalized feature vector corresponding to each frequency point, and calculating the projection energy of the normalized feature vector corresponding to each frequency point on the normalized feature vector corresponding to each predetermined posture. The processing process of the posture matching detection module may include: performing weighted addition on the projection energy of all frequency points to obtain the total projection energy of each predetermined posture, selecting the predetermined posture corresponding to the maximum projection energy as the user posture, and obtaining the posture detection result. The above-mentioned two modules in the posture detection module will be described in detail below.
[0127] In an optional embodiment of the present application, obtaining the projection energy of the speech signal corresponding to each predetermined posture includes:
[0128] Obtaining the projection energy of each frequency point of the speech signal corresponding to each predetermined posture;
[0129] Get the weight value corresponding to each frequency point;
[0130] The projection energy of the speech signal corresponding to each predetermined posture is determined according to the weight values corresponding to each frequency point and the projection energy of each frequency point corresponding to each predetermined posture.
[0131] Among them, the projection energy calculation module obtains the projection energy of each frequency point of the voice signal corresponding to each predetermined posture, and the posture matching detection module obtains the weight value corresponding to each frequency point, determines the projection energy of the voice signal corresponding to each predetermined posture, and further determines the user's posture, that is, obtains the posture detection result.
[0132] Specifically, for obtaining the projection energy of the speech signal corresponding to each predetermined posture, the general processing process is: obtaining at least two frequency domain signals corresponding to the speech signal; for each frequency point, merging the eigenvalues of each frequency domain signal at the frequency point to obtain the eigenvector of the speech signal. Obtaining the eigenvector corresponding to the speech signal, the eigenvector contains the eigenvalues corresponding to each frequency point; normalizing the eigenvector to obtain the normalized eigenvector corresponding to the speech signal; determining the projection energy of each frequency point corresponding to each predetermined posture according to the normalized eigenvector and the eigenmatrix corresponding to each predetermined posture.
[0133] It should be noted that the unnormalized eigenvectors of each frequency point and the eigenmatrices corresponding to each predetermined posture may also be used to determine the projection energy of each frequency point corresponding to each predetermined posture, which is not limited in the embodiments of the present application.
[0134] Furthermore, for each frequency point, the eigenvalues of each frequency domain signal at the frequency point are combined to obtain the eigenvector corresponding to the frequency point, and then based on the eigenvector of the frequency point and the eigenmatrix corresponding to each predetermined posture at the frequency point, the projection energy of each predetermined posture corresponding to the frequency point is determined. In the above manner, the projection energy of all frequency points of the speech signal under each predetermined posture can be determined.
[0135] Taking the example of two voice collection devices being set on a call device to illustrate the acquisition of projection energy at each frequency point, the two voice collection devices are an upper microphone and a lower microphone, such as Figure 8 As shown, first, the speech signal s1 collected by the upper microphone and the speech signal s2 collected by the lower microphone are framed, windowed and short-time Fourier transformed, where the window length is L, and the corresponding two frequency domain signals are recorded as x1 k and x2 k , take the values of the same frequency points (frequency points) of the two signals to form a vector X k =[x1 k ,x2 k ], which is the characteristic vector of the frequency point, where Represents the frequency point of 0Hz to 8000Hz. Then, the feature vector of each frequency point is normalized, that is, only the phase information is focused on and the amplitude information is ignored. This can avoid the influence of the user's voice on the posture detection. The expression of the normalization process is: Among them, X′ k For X kFinally, based on the normalized eigenvector corresponding to each frequency point and the eigenmatrix of each predetermined posture at the frequency point, the projection energy corresponding to each frequency point is calculated: Here (j,k) is the projection energy corresponding to frequency point k at posture j, C (j,k) is the feature matrix of the jth predetermined posture at frequency point k.
[0136] In an optional embodiment of the present application, before normalizing the feature vector, the method further includes:
[0137] According to the predetermined compensation parameters, the frequency response compensation processing is performed on the eigenvector to obtain the eigenvector after amplitude correction.
[0138] Specifically, the expression for amplitude correction of the characteristic vector of each frequency point is: k =[1,r k ]′⊙X k , where ⊙ represents the multiplication of the corresponding points of the vector, r k is a fixed value, representing the ratio of the frequency responses of different microphone hardware to a certain frequency. This step is to correct the differences between different microphone hardware. For example, if at a certain frequency point k, the frequency response of the upper microphone is 2 and the frequency response of the lower microphone is 4, then r k The value of can be 2 / 4 = 0.5, so X k =[1,0.5]′⊙[2,4]′=[2,2], it can be seen that the amplitudes of different microphones are corrected to the same level.
[0139] In an optional embodiment of the present application, for at least one predetermined posture, the corresponding feature matrix is obtained in the following manner:
[0140] Determine the distance between the sample sound source and each voice collection device respectively;
[0141] Determine a feature vector corresponding to a predetermined posture according to the distance between the sample sound source and each voice collection device;
[0142] Based on the feature vector corresponding to the predetermined posture, a feature matrix corresponding to the predetermined posture is determined.
[0143] Among them, the characteristic matrix corresponding to each predetermined posture at each frequency point is the autocorrelation matrix of the characteristic vector corresponding to the predetermined posture. Converting the characteristic vector of each predetermined posture at each frequency point into an autocorrelation matrix (i.e., characteristic matrix) can more conveniently calculate the projection energy corresponding to each frequency point.
[0144] Specifically, the feature matrix C of each predetermined posture (j,k) The calculation process is as follows Fig. 9As shown, first, a plurality of predetermined postures are preset. Generally speaking, the predetermined posture range is [-30°~60°]. Other angles are not commonly used in handheld mode calls. The selection of the number P of predetermined postures is a balance between complexity and performance. The embodiment of the present application takes P=4 as an example, which means j=0, 1, 2, 3. (The number of P is variable and can be selected according to actual needs). Then, for each predetermined posture, the distance d1 from the user's mouth to the upper microphone is measured respectively. j and the distance d2 to the lower microphone j , where the subscript j represents different predetermined postures. Then, the normalized eigenvectors corresponding to each predetermined posture are calculated. Each predetermined posture j has a corresponding eigenvector at each frequency point k, and its expression is: Here A (j,k) is a 2*1 vector, s represents the speed of sound (343m / s). Finally, calculate the feature matrix corresponding to each predetermined posture at frequency point k: C (j,k) =A (j,k) ·′ (j,k) , C (j,k) It is a 2*2 matrix.
[0145] Further, for the case where the terminal device includes three microphones, for example, the three microphones are respectively located above, below and to the side of the terminal device, for each predetermined posture, after obtaining the distance between the sound source and each microphone, the relative distance between the sound source (user's mouth) and each microphone is calculated, wherein the relative distance value between the sound source and each microphone is:
[0146] [0,d2 j -d1 j ,d3 j -d1 j ]
[0147] Among them, d1 j is the distance from the microphone to the sound source at the predetermined posture j, d1 j is the distance between the sound source and the side microphone in the predetermined posture j, d1 j is the distance between the sound source and the upper microphone in the predetermined posture j. For example, for posture 2, the sound source is 6 cm away from the lower microphone, 10 cm away from the side microphone, and 12 cm away from the upper microphone. The relative distance is [0, 4, 6].
[0148] Furthermore, for each predetermined posture, after calculating the relative distance between the sound source and each microphone, its feature vector at each specific frequency point is calculated, that is, the feature vector corresponding to the posture at each specific frequency point:
[0149]
[0150] Among them, A (j,k) is a 3*1 vector.
[0151] After obtaining the projection energy of each frequency point of the voice signal corresponding to each predetermined posture in the projection energy calculation module, the weight value corresponding to each frequency point is obtained in the posture matching detection module to determine the projection energy of the voice signal corresponding to each predetermined posture, and further determine the posture of the user, that is, to obtain the posture detection result. Specifically, there are two ways to obtain the weight value corresponding to each frequency point in the posture matching detection module. One is to obtain the predetermined weight corresponding to each frequency point, and the other is to determine the weight corresponding to each frequency point through a weight determination neural network based on the projection energy of each frequency point corresponding to each predetermined posture and / or the energy information of the voice signal.
[0152] Specifically, when the weight corresponding to each frequency point is a predetermined weight, the posture detection process of the posture detection module in the embodiment of the present application is as follows: Fig.10 As shown. The predetermined weight corresponding to each frequency point can be w = [0, 0, ..., 0, 1, 1, ..., 1, 0, 0, ..., 0], the number of 0s at the front is M, the number of 1s in the middle is N, and the number of 0s at the end is Z. Among them, M = floor (800 * L / fs), N = floor (4000 * L / fs) - M, Z = floor (L / 2) + 1 - MN; L is the window length, fs is the sampling frequency of the voice acquisition device, that is, only the frequency point weight of 800 ~ 4000Hz is set to 1, and the others are set to 0, because 800 ~ 4000Hz contains the most voice components, while other frequencies are relatively more noise components. The weight corresponding to each frequency point adopts a predetermined weight, which can reduce the computational complexity of the posture detection process.
[0153] Specifically, when the neural network is determined by weights to determine the weights corresponding to each frequency point, the posture detection process of the posture detection module in the embodiment of the present application is as follows: Fig.11 As shown. Among them, the energy of the speech signal at each frequency point is calculated: E k =Z k ·X′ k , and the projection energy e(j, k) of each frequency point of the speech signal output by the projection energy calculation module corresponds to each predetermined posture and the energy E of each frequency point k Input to the weight determination neural network, output the weight w corresponding to each frequency point k .
[0154] The reason for calculating different weights for different frequency points is as follows: each frequency point corresponds to different speech components, some frequency points contain more noise components, and some frequency points contain more speech components. Obviously, the frequency points with more speech components are more important to gesture detection, and they will be given a greater weight. Conversely, the frequency points with more noise components have a lower weight. This method of calculating different weights for each frequency point can improve the accuracy of gesture detection in a strong noise environment.
[0155] Furthermore, after obtaining the weights corresponding to each frequency point through the above two weight acquisition methods, for each predetermined posture, the projection energy of each frequency point corresponding to the predetermined posture is weighted according to the weight value corresponding to each frequency point; for each predetermined posture, the weighted projection energy of each frequency point corresponding to the predetermined posture is accumulated to obtain the projection energy of the speech signal corresponding to the predetermined posture. Its expression can be e j =∑ k w k ·e (j,k) When the speech signal contains 257 frequency points (k = 0 ~ 256), e j The calculation is as follows Fig.12 shown.
[0156] Finally, the predetermined posture corresponding to the maximum posture energy information is selected to obtain the posture information of the user during the voice call. j ={e 0 ,e 1 ,…e P-1}, the corresponding subscript is the final detection result, that is, the current terminal device posture: h index = argmax(e j ), where index = {0, 1, .., P-1}.
[0157] In an optional embodiment of the present application, the weight determination neural network includes a control subnetwork and a calculation subnetwork; through the control subnetwork, the signal-to-noise ratio characterization value of the speech signal is determined according to the energy information of the speech signal, and whether the weight value of each frequency point is a predetermined weight is determined according to the signal-to-noise ratio characterization value; if not, through the calculation subnetwork, the weight value of each frequency point is determined according to the projection energy of each frequency point corresponding to each predetermined posture.
[0158] Among them, the control subnetwork and the computing subnetwork are connected through the connection layer, such as Fig.13 The figure shows a schematic diagram of the structure of a weight determination neural network in an example, in which four predetermined postures are pre-set, and the sampling rate of the two voice acquisition devices is 16kHz, and the window length is 512 frequency points. Among them, the input layer of the control subnetwork can contain 514 nodes, specifically the energy E of each frequency point in the voice signalk (two microphone signals, upper and lower, corresponding to 257 frequency points), the output layer contains 1 node, specifically the control flag f (i.e., signal-to-noise ratio representation value). The input layer of the computing subnetwork contains 1258 nodes, specifically the projection energy e(j,k) of 1028 frequency points (4 predefined postures, corresponding to 257 frequency points in the frequency domain), and the output layer contains 257 nodes, specifically the weights w of 257 frequency points k .
[0159] Specifically, if the signal-to-noise ratio representation value f output by the control subnetwork is not less than the preset threshold, the calculation of the calculation subnetwork can be skipped, and the weight of each frequency point is directly reset to 1 (w = [1, 1, ..., 1]). If the total heart rate representation value f output by the control subnetwork is less than the preset threshold, it is included in the calculation subnetwork to calculate the weight of each frequency point. Among them, the preset threshold can be set to 0.5.
[0160] It should be noted that the purpose of adding a control subnetwork to the design of the weight determination neural network is to reduce the computational complexity of the neural network, thereby saving power consumption of the device. Because in some cases, the environment is very quiet (such as home, office, etc.), the background noise has little impact, and there is no need to calculate the precise weight w corresponding to each frequency band to obtain better posture detection results. Setting w to [1,1,…,1] is sufficient to detect the accurate position. Then, during the operation of the weight determination neural network, the signal-to-noise ratio representation value is output by the control subnetwork to judge the size of the environmental noise. When the environmental noise is small (that is, the signal-to-noise ratio representation value is not less than the preset threshold), the calculation subnetwork is skipped and the weight of each frequency point is directly reset to 1, which can greatly reduce the computational complexity. Of course, if the computational complexity is not considered, the weight determination neural network can only include the calculation subnetwork, and the weight of each frequency point can also be obtained.
[0161] In an optional embodiment of the present application, the control subnetwork extracts features of energy information of the speech signal via a plurality of cascaded first feature extraction layers, and obtains a signal-to-noise ratio representation value according to the extracted features via a classification layer;
[0162] The computing subnetwork extracts features of the projection energy of each frequency point corresponding to each predetermined posture through a plurality of cascaded second feature extraction layers, and obtains the weight of each frequency point according to the extracted features through a linear regression layer.
[0163] The second feature extraction layer merges the extracted features with the features output by the corresponding first feature extraction layer in the control subnetwork, and outputs the merged features.
[0164] Specifically, refer again to Fig.13In the weight determination neural network, the control subnetwork includes a control subnetwork input layer 1201, two first feature extraction layers (i.e., fully connected layers) 1202 and 1203, and a control subnetwork output layer (i.e., classification layer) 1204. Specifically, the control subnetwork input layer 1201 is used to receive the energy of each input frequency point, the first feature extraction layer 1202 is used to extract corresponding low-level features from the energy of each input frequency point, the first feature extraction layer 1203 is used to extract corresponding high-level features from the energy of each frequency point based on the low-level features input by the first feature extraction layer 1202, and the control subnetwork output layer 1204 is used to output corresponding signal-to-noise ratio representation values based on the high-level features input by the first feature extraction layer 1203. The weight determination neural network includes a computing subnetwork input layer 1205, three second feature extraction layers (i.e., fully connected layers) 1206, 1208, and 1210, two connection layers 1207 and 1209, and a computing subnetwork output layer 1211. Specifically, the computing subnetwork input layer 1205 is used to receive the projection energy of each input frequency point, the second feature extraction layer 1206 is used to extract the corresponding low-level features from the projection energy of each input frequency point, and the connection layer 1207 is used to connect the first feature extraction layer 120 2 and the low-level features extracted by the second feature extraction layer 1206, the second feature extraction layer 1208 is used to extract the corresponding high-level features based on the two low-level features input by the connection layer 1207, the connection layer 1209 is used to connect the high-level features extracted by the first feature extraction layer 1203 and the high-level features extracted by the second feature extraction layer 1208, and the second feature extraction layer 1210 is used to determine the weight value of each frequency point based on the two high-level features input by the connection layer 1209, and output it through the calculation subnetwork output layer 1211. Among them, the corresponding number of nodes and activation functions of each network layer in this example are listed in the following Table 3.
[0165] Table 3
[0166]
[0167]
[0168] It should be noted that the network layer structure of the above-mentioned weight determination neural network is only an example. In practical applications, the weight determination neural network can also adopt CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), ResNet (Residual Network) or other existing neural networks.
[0169] It is understandable that before using the weight determination neural network to obtain the weights of each frequency point, the weight determination neural network needs to be trained. In the process of training the weight determination neural network, since the weight determination neural network contains a control subnetwork and a calculation subnetwork, there are two labels used in the training, namely, the output signal-to-noise ratio representation value of the control subnetwork Calculate the weight value of each frequency point of the sub-network output Specifically, first obtain a speech signal sample, and then obtain the noise ratio representation value of the speech signal sample. And the weight value of each frequency point in the speech signal sample That is, we get the sample data for training. The expression of the loss function used in the training process is as follows:
[0170]
[0171] The noise ratio representation value for the speech signal sample If the SNR corresponding to the speech signal sample is greater than 10dB, it can be considered that the energy of the user's speech contained in the speech sample is much greater than the energy of the noise. At this time, it means that the user is in a quiet environment. Set to 1. If the SNR corresponding to the speech signal sample is not greater than 10dB, it can be considered that the energy of the user's speech contained in the speech sample is not much greater than the energy of the noise. This means that the user is in a noisy environment. Set to 0. Specifically, Use the following expression to calculate:
[0172]
[0173] Assuming that the speech signal sample contains 257 frequency points (k=0-256), the calculation formula of SNR (Signal Noise Ratio) is:
[0174]
[0175] Among them, Y k is the energy of the user's speech contained in frequency point k, N k is the energy of the noise contained in frequency point k.
[0176] The weight value of each frequency point in the speech signal sample Assuming that 4 predetermined postures (j=0-3) are preset, if the projection energy e corresponding to the frequency point k (0,k) ,e (1,k) ,e (2,k) ,e (3,k)It is more important to obtain more accurate posture results, so it is necessary to give a higher weight to the projection energy corresponding to the frequency point k Specifically, it can be calculated in two steps
[0177] Step 1: Calculate the initial weight value of frequency point k according to the following formula (ensuring that the frequency points with less noise have larger initial weight values)
[0178]
[0179] Among them, Y k is the energy of the user's speech contained in frequency point k, E k is the energy of the speech signal at frequency point k.
[0180] Specifically, assuming that the speech signal sample contains 257 frequency points (k=0-256) and 4 predetermined postures (j=0-3) are set, then the initial weight values of each frequency point are:
[0181] Step 2: According to the magnitude of the projection energy of the frequency point k in each predetermined posture, the initial weight value of the frequency point k is adjusted.
[0182] Specifically, assuming that the second posture (i.e., j=1) in the predetermined posture is the target posture (i.e., the actual posture of the user), if e (1,k) For (0,k) ,e (1,k) ,e (2,k) e (3,k) ] is the largest, then [e (0,k) ,e (1,k) ,e (2,k) e (3,k) ] is more important for obtaining more accurate posture results, so the weight value of frequency point k is changed from the initial weight value Adjust to like Fig.14 The adjustment in this example is shown.
[0183] For example, for step 1, Fig.15 As shown, if k = 2, Y 2 =6,N 2 =2,E 2 =8, so If k = 3, Y 3 =3,N 3 =7,E 3 =10, so in, This indicates that since the voice energy proportion of the user in frequency point 2 (6 / 8) is greater than the voice energy proportion of the user in frequency point 3 (3 / 10), frequency point 2 is more important when calculating the weighted projection energy under each posture in the posture matching detection module. It further indicates that in frequency point k, the greater the proportion of the user's voice energy, the greater the corresponding initial weight value. This is mainly because if the proportion of the user's voice energy in a frequency point is higher, the noise impact in the frequency point will be smaller, and the posture detection result corresponding to the frequency point will be more accurate, so a higher initial weight value should be given to the frequency point. Otherwise, a lower weight value should be given to reduce the impact of noise.
[0184] The calculation formula of the initial weight value is used to limit the value range of the initial weight value to 0-1, so as to eliminate some abnormal values that appear in the encoding process.
[0185] The calculation formula for the initial weight value is The reason is that in the next step, the initial weight value of the frequency point that meets the conditions will be multiplied by 2 to obtain the final weight label. This ensures that the value range of the final weight label is 0 to 1. Since the weight determination neural network outputs the weight value of each frequency point through the activation function Sigmoid, and the value range of the output of the activation function Sigmoid is 0 to 1, it is necessary to limit the value range of the seed weight label to 0 to 1.
[0186] For step 2, assuming that the actual user posture is the predetermined posture q, for frequency point k, if e (q,k) for e (j,k) ,j=0,1,2,3, the largest, then the weight value of the frequency point k should be larger. (q,k) The maximum value indicates that the frequency point k is more important for detecting the real user posture, and a higher weight value should be given to the frequency point k.
[0187] For example, assuming that the actual user posture is the second predetermined posture (i.e., q=1), and to illustrate. Fig.16a In the figure, we can see that e (1,2) The largest, therefore That is, the weight value of frequency point 2 is adjusted to twice the initial weight value. Fig.16b In, e (0,3) The largest, and e (1,3) If it is not the maximum, then That is, the weight value of frequency point 3 is kept unchanged as the initial weight value.
[0188] To simplify the above example, it is assumed that the speech signal sample contains only two frequency points (k=2,3), and the weight value of the neural network output is the same as the label.
[0189] Then, if the initial weight value is not adjusted in step 2, and the weight value obtained in step 1 is directly used to project the energy e under each predetermined posture, j The calculation results are as follows:
[0190] e 0 =e (0,2) ×0.375+e (0,3) ×0.15=0.285
[0191] e 1 =e (1,2) ×0.375+e (1,3) ×0.15=0.27
[0192] e 2 =e (2,2) ×0.375+e (2,3) ×0.15=0.143
[0193] e 3 =e (3,2) ×0.375+e (3,3) ×0.15=0.225
[0194] If the weight value adjusted in step 2 is used for e j The calculation results are as follows:
[0195] e 0 =e (0,2) ×0.75+e (0,3) ×0.15=0.435
[0196] e 1 =e (1,2) ×0.75+e (1,3) ×0.15=0.495
[0197] e 2 =e (2,2) ×0.75+e (2,3) ×0.15=0.255
[0198] e 3 =e (3,2) ×0.75+e (3,3) ×0.15=0.375
[0199] From the above calculation results, it can be seen that without using step 2, e 0 The maximum, the posture detection result is the first predetermined posture, which is incorrect; while in the case of step 2, e 1The maximum,posture detection result is the second predetermined posture, which is correct, so adopting step 2 can improve the accuracy of posture detection.
[0200] It should be noted that the posture detection method provided in the embodiment of the present application is applicable not only to the case where two voice collection devices are included in the above example, but also to the case where three or more voice collection devices are included. Fig.17 As shown, many existing terminal devices are equipped with three voice collection devices. The three voice collection devices of the terminal device shown on the left side of the figure are respectively a top microphone, a bottom microphone and a back microphone, and the three voice collection devices of the terminal device shown on the right side of the figure are respectively a top microphone and two bottom microphones (bottom microphone 1 and bottom microphone 2).
[0201] For the case involving three or more voice collection devices, the corresponding posture detection method principle is similar to the above description. Specifically, the implementation process of posture detection can be as follows: Fig.18 As shown in the figure, the processing process in the posture matching detection module is the same as that in the case of two voice acquisition devices, and the processing process in the projection energy calculation module is slightly different. Specifically, in the projection energy calculation module, first, the voice signals (s1, s2, ..., sM) collected by multiple voice acquisition devices (microphones) are framed, windowed, and short-time Fourier transformed, where the window length is L, to obtain the multi-channel frequency domain signal x1 k ,x2 k ,…,xn k ,…,xM k , take the values of the same frequency points of multiple signals to form a vector X k =[x1 k ,x2 k ,…,xn k ,…,xM k ], that is, the eigenvector of the frequency point, where represents the frequency point from 0Hz to 8000Hz. Then, the amplitude of the characteristic vector of each frequency point is corrected: X k =[1,…,r1n k …,1M k ]′⊙X k , where ⊙ represents the multiplication of the corresponding points of the vector, r1 k is a fixed value, representing the ratio of the frequency response of different microphone hardware to a certain frequency. This step is to correct the differences between different microphone hardware. For example, if at a certain frequency point k, the frequency response of microphone 1 is 2 and the frequency response of microphone 2 is 4, then r1 k The value should be 0.5. Then, the eigenvectors of each frequency point are normalized, that is, only the phase information is focused on and the amplitude information is ignored. The normalized expression is Among them, X′ k For X k Finally, the normalized eigenvector corresponding to each frequency point and the eigenmatrix of each predetermined posture are used to calculate the projection energy corresponding to each frequency point: Here (j,k) is the projection energy corresponding to frequency point k at posture j.
[0202] For a case where there are three or more voice collection devices, the process of obtaining the feature matrix of the predetermined posture may include: first, for each predetermined posture, measuring the distance d1 from the user's mouth to the first microphone respectively. j , the distance to the second microphone d2 j , and the distance dn to the nth mic j , where the subscript j represents different postures. Then, the normalized feature vector corresponding to each predetermined posture is calculated. For each posture j and each frequency point k, there is a corresponding feature vector, and its expression is: Here A (j,k) is an M*1 vector, s represents the speed of sound (343m / s). Finally, calculate the feature matrix corresponding to each predetermined posture: C (j,k) =A (j,k) ·′ (j,k) , C (j,k) It is a matrix of size M*M.
[0203] The following is a test example to illustrate the beneficial effects of the posture detection method provided in the embodiment of the present application. It is assumed that the terminal device used by the user is Fig.17 The terminal device shown on the left includes three microphones, namely the top microphone (corresponding to Fig.19 The upper mic in the middle), the bottom microphone (corresponding to Fig.19 The lower mic) and the back microphone (corresponding to Fig.19 ), assuming that the projection energy calculation module is pre-set at least as follows Figure 5a-5c There are 4 predefined postures in the test process, such as Fig.19 As shown, the upper part of the figure shows the waveforms of the voice signals collected by the three microphones respectively, which are divided into four sections, each section corresponds to a different signal-to-noise ratio (from left to right: high signal-to-noise ratio, medium signal-to-noise ratio, low signal-to-noise ratio and very low signal-to-noise ratio), and each section is divided into four small sections, corresponding to the four postures of the user (from left to right: predetermined posture 1, predetermined posture 2, predetermined posture 3 and predetermined posture 4). The posture detection method of the embodiment of the present application is used to determine the posture of the user during the voice call based on the voice signals collected by the above three microphones. The detection results are shown in FIG. Fig.19As shown in the lower part, it can be seen that even in low signal-to-noise ratio or extremely low signal-to-noise ratio conditions, high detection accuracy can still be maintained.
[0204] Therefore, by guiding the multi-microphone voice enhancement module with the posture detection method provided in the embodiment of the present application, it is also possible to avoid performance degradation caused by the inability to judge the user's posture in a high-noise or extremely high-noise environment, so as to maximize the advantages of multiple microphones and obtain better call voice enhancement effects.
[0205] The following example illustrates gesture detection and call voice quality enhancement in a voice call scenario. Fig. 20 As shown, the near-end speaker (i.e., the user in the embodiment of the present application) establishes a call link with the far-end caller. After the near-end speaker utters a voice, "time-frequency conversion" is first performed to convert the voice signals collected by multiple microphones from time domain to frequency domain; then, the posture of the near-end speaker's handheld device is detected through the frequency domain spectrum of the multiple microphone signals. The speaker posture detection in the dotted box in the figure adopts the posture detection method provided in the embodiment of the present application; then, the detection result is used to guide the call voice enhancement module to perform call voice enhancement and output the enhanced single-channel voice spectrum; then, the enhanced voice spectrum is "frequency-time conversion" is performed to convert the enhanced voice spectrum to the time domain; finally, the clean voice is transmitted to the far-end caller.
[0206] At the same time, an embodiment of the present application also provides another posture detection method. During a voice call, the distance between the user's mouth and different microphones is also different, which means that the time delay at different microphones is also different. This feature can be used to detect the user's posture during a voice call.
[0207] like Fig.21 As shown, the above-mentioned scenario involving two microphones is used as an example to explain the solution in detail. First, the upper and lower microphones collect speech signals s1 and s2 respectively; then, the correlation signal R between the speech signals s1 and s2 is calculated. s1s2 (τ), the calculation expression is:
[0208] R s1s2 (τ)=E[s1(t)*s2(t-τ)]
[0209] Where τ = [-T, T], T = 0.01s; then, the time delay is calculated by the peak value of the correlation signal = argmax(R S1S2 (τ)), and finally, the user posture is matched based on the event delay to complete the posture detection.
[0210] This posture detection method is suitable for quiet environments. Since it uses the time delay of the user's clean voice to estimate the posture, when the noise is large, it will affect the accuracy of the time delay estimation, thereby affecting the accuracy of posture detection. This method is suitable for quiet environments (i.e., environments with high signal-to-noise ratios), so a high-performance and high-reliability noise classification module or signal-to-noise ratio estimation module can be placed before the posture detection module to determine whether the current environment meets the posture detection requirements.
[0211] Fig. 22 A structural block diagram of a posture detection device provided in an embodiment of the present application, such as Fig. 22 As shown, the device 1700 may include: a voice collection module 1701, an energy acquisition module 1702 and a posture determination module 1703, wherein:
[0212] The voice collection module 1701 is used to collect voice signals during a voice call through at least two voice collection devices;
[0213] The energy acquisition module 1702 is used to obtain the gesture energy information corresponding to the voice signal;
[0214] The posture determination module 1703 is used to determine the posture of the user during the voice call from various predetermined postures according to the posture energy information.
[0215] By pre-setting multiple predetermined postures, the feature vectors of voice signals collected by multiple voice collection devices during the voice call are used as the feature vectors of the user posture, and by comparing the feature vectors of the user posture with the feature vectors of each predetermined posture respectively, the predetermined posture corresponding to the closest feature vector is determined as the user posture, that is, the user posture detection result is obtained, which can provide a reference for enhancing the call voice quality and optimize the effect of noise suppression processing.
[0216] In an optional embodiment of the present application, the energy acquisition module is specifically used for:
[0217] The projection energy of the speech signal corresponding to each predetermined posture is obtained.
[0218] In an optional embodiment of the present application, the energy acquisition module further includes a first projection energy acquisition submodule, a weight acquisition submodule and a second projection energy acquisition submodule, wherein:
[0219] The first projection energy acquisition submodule is used to acquire the projection energy of each frequency point of the voice signal corresponding to each predetermined posture;
[0220] The weight acquisition submodule is used to obtain the weight values corresponding to each frequency point;
[0221] The second projection energy acquisition submodule is used to determine the projection energy of the speech signal corresponding to each predetermined posture according to the weight value corresponding to each frequency point and the projection energy of each frequency point corresponding to each predetermined posture.
[0222] In an optional embodiment of the present application, the first projection energy acquisition submodule is specifically used for:
[0223] Obtain the eigenvector corresponding to the speech signal, where the eigenvector contains the eigenvalues corresponding to each frequency point;
[0224] Normalizing the feature vector to obtain a normalized feature vector corresponding to the speech signal;
[0225] The projection energy of each frequency point corresponding to each predetermined posture is determined according to the normalized eigenvector and the eigenmatrices corresponding to each predetermined posture.
[0226] In an optional embodiment of the present application, the first projection energy acquisition submodule is further used for:
[0227] Acquire at least two frequency domain signals corresponding to the speech signal;
[0228] For each frequency point, the eigenvalues of each frequency domain signal at that frequency point are combined to obtain the eigenvector of the speech signal.
[0229] In an optional embodiment of the present application, the device further includes an amplitude correction module, which is used to: before normalizing the eigenvector, perform frequency response compensation processing on the eigenvector according to a predetermined compensation parameter to obtain an amplitude-corrected eigenvector.
[0230] In an optional embodiment of the present application, the device further includes a feature matrix acquisition module, which is used to:
[0231] Determine the distance between the sample sound source and each voice collection device respectively;
[0232] Determine a feature vector corresponding to a predetermined posture according to the distance between the sample sound source and each voice collection device;
[0233] Based on the feature vector corresponding to the predetermined posture, a feature matrix corresponding to the predetermined posture is determined.
[0234] In an optional embodiment of the present application, the weight acquisition submodule is specifically used for:
[0235] Get the predetermined weight corresponding to each frequency point.
[0236] In an optional embodiment of the present application, the weight acquisition submodule is specifically used for:
[0237] According to the projection energy of each frequency point corresponding to each predetermined posture and / or the energy information of the voice signal, a neural network is determined through weights to determine the weights corresponding to each frequency point.
[0238] In an optional embodiment of the present application, the weight determination neural network includes a control subnetwork and a calculation subnetwork; the weight acquisition submodule is further used to:
[0239] Determine, by the control subnetwork, a signal-to-noise ratio characterization value of the speech signal according to energy information of the speech signal, and determine, according to the signal-to-noise ratio characterization value, whether the weight value of each frequency point is a predetermined weight;
[0240] If not, the weight value of each frequency point is determined by the calculation sub-network according to the projection energy of each frequency point corresponding to each predetermined posture.
[0241] In an optional embodiment of the present application, the control subnetwork extracts features of energy information of the speech signal via a plurality of cascaded first feature extraction layers, and obtains a signal-to-noise ratio representation value according to the extracted features via a classification layer;
[0242] The computing subnetwork extracts features of the projection energy of each frequency point corresponding to each predetermined posture through a plurality of cascaded second feature extraction layers, and obtains the weight of each frequency point according to the extracted features through a linear regression layer.
[0243] In an optional embodiment of the present application, the second feature extraction layer merges the extracted features with the features output by the corresponding first feature extraction layer in the control subnetwork, and outputs the merged features.
[0244] In an optional embodiment of the present application, the second projection energy acquisition module is specifically used for:
[0245] For each predetermined posture, the projection energy of each frequency point corresponding to the predetermined posture is weighted according to the weight value corresponding to each frequency point;
[0246] For each predetermined posture, the weighted projection energies of each frequency point corresponding to the predetermined posture are accumulated to obtain the projection energy of the speech signal corresponding to the predetermined posture.
[0247] In an optional embodiment of the present application, the posture determination module is specifically used for:
[0248] The predetermined posture corresponding to the maximum posture energy information is selected to obtain the posture information of the user during the voice call.
[0249] In an optional embodiment of the present application, the device may include a noise reduction module, which is used to:
[0250] Based on the user's posture information during the voice call, the voice signal is denoised to obtain a noise-reduced voice signal.
[0251] Reference below Fig.23 , which shows an electronic device suitable for implementing the embodiments of the present application (for example, executing Figure 4 The electronic device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.23 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0252] The electronic device includes: a memory and a processor, the memory is used to store a program for executing the method described in each of the above method embodiments; the processor is configured to execute the program stored in the memory. The processor here can be referred to as the processing device 1801 described below, and the memory can include at least one of the read-only memory (ROM) 1802, the random access memory (RAM) 1803, and the storage device 1808 described below, as shown below:
[0253] like Fig.23 As shown, the electronic device 1800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1802 or a program loaded from a storage device 1808 to a random access memory (RAM) 1803. In the RAM 1803, various programs and data required for the operation of the electronic device 1800 are also stored. The processing device 1801, the ROM 1802, and the RAM 1803 are connected to each other via a bus 1804. An input / output (I / O) interface 1805 is also connected to the bus 1804.
[0254] Typically, the following devices may be connected to the I / O interface 1805: an input device 1806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1809. The communication device 1809 may allow the electronic device 1800 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.23An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0255] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1809, or installed from the storage device 1808, or installed from the ROM 1802. When the computer program is executed by the processing device 1801, the above-mentioned functions defined in the method of the embodiment of the present application are executed.
[0256] It should be noted that the computer-readable storage medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0257] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0258] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0259] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:
[0260] The voice signal during the voice call is collected by at least two voice collection devices; the posture energy information corresponding to the voice signal is obtained; and the posture of the user during the voice call is determined from various predetermined postures according to the posture energy information.
[0261] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0262] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0263] The modules or units involved in the embodiments of the present application may be implemented by software or hardware. The name of a module or unit does not limit the unit itself in some cases. For example, a voice collection module may also be described as a "module for collecting voices".
[0264] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0265] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0266] The device provided in the embodiment of the present application can implement at least one module among the multiple modules through an AI model. Functions associated with AI can be executed through a non-volatile memory, a volatile memory, and a processor.
[0267] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).
[0268] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.
[0269] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device itself in which the AI according to the embodiment is executed, and / or can be implemented by a separate server / system.
[0270] The AI model may include multiple neural network layers. Each layer has multiple weight values, and the calculation of one layer is performed by the calculation result of the previous layer and the multiple weights of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.
[0271] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.
[0272] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific method implemented when the above-described computer-readable medium is executed by an electronic device can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.
Claims
1. A method performed by an electronic device, It is characterized in that include: Collecting voice signals during a voice call by at least two voice collection devices; Acquiring position energy information of the speech signal; According to the position energy information, the position of the electronic device relative to the user during the voice call is determined from multiple predetermined positions; the predetermined position is set corresponding to the angle between the central axis of the electronic device and the central axis of the user's face; the angle includes multiple ones.
2. The method according to claim 1, It is characterized in that Acquiring the position energy information of the speech signal, including: The projection energy of the speech signal corresponding to each predetermined position among the plurality of predetermined positions is obtained.
3. The method according to claim 2, It is characterized in that Acquiring the projection energy of the speech signal corresponding to each predetermined position in the plurality of predetermined positions, comprising: Acquire projection energy of each frequency point among a plurality of frequency points corresponding to each predetermined position among the plurality of predetermined positions, wherein the plurality of frequency points are included in the speech signal; Obtaining a weight value corresponding to each frequency point in the multiple frequency points; The projection energy of the speech signal corresponding to each of the multiple predetermined positions is determined according to the weight value corresponding to each of the multiple frequency points and the projection energy of each of the multiple frequency points corresponding to each of the multiple predetermined positions.
4. The method according to claim 3, It is characterized in that The acquiring the projection energy of each frequency point among the multiple frequency points corresponding to each predetermined position among the multiple predetermined positions includes: Acquire multiple feature vectors corresponding to the speech signal, wherein the feature vectors include feature values corresponding to each frequency point in the multiple frequency points; Normalizing the multiple feature vectors to obtain a normalized feature vector corresponding to the speech signal; The projection energy of each frequency point among the multiple frequency points corresponding to each predetermined position among the multiple predetermined positions is determined according to the normalized feature vector and the feature matrix corresponding to each predetermined position among the multiple predetermined positions.
5. The method according to claim 4, It is characterized in that Acquiring a plurality of feature vectors corresponding to the speech signal, including: Acquire at least two frequency domain signals corresponding to the speech signal; The eigenvalues of at least two frequency domain signals in the multiple frequency points are combined to obtain multiple eigenvectors of the speech signal.
6. The method according to claim 4, It is characterized in that Before normalizing the feature vector, the method further includes: According to predetermined compensation parameters, frequency response compensation processing is performed on the multiple eigenvectors to obtain amplitude-corrected eigenvectors.
7. The method according to claim 4, It is characterized in that For at least one predetermined position among the plurality of predetermined positions, the corresponding feature matrix is obtained in the following manner: Determine the distance between the sample sound source and each voice collection device respectively; Determining feature vectors corresponding to the plurality of predetermined positions according to the distances between the sample sound source and each voice collection device; Based on the feature vectors corresponding to the multiple predetermined positions, a feature matrix corresponding to the multiple predetermined positions is determined.
8. The method according to claim 3, It is characterized in that Obtaining a weight value corresponding to each frequency point in the plurality of frequency points, including: Obtain a predetermined weight corresponding to each frequency point in the multiple frequency points.
9. The method according to claim 3, It is characterized in that Obtaining a weight value corresponding to each frequency point in the plurality of frequency points, including: According to the projection energy of each frequency point in the multiple frequency points corresponding to each predetermined position in the multiple predetermined positions and / or the energy information of the speech signal, a weight corresponding to each frequency point in the multiple frequency points is determined through a weight determination neural network.
10. The method according to claim 9, It is characterized in that The weight determination neural network includes a control subnetwork and a calculation subnetwork; Determining a weight corresponding to each frequency point in the plurality of frequency points by using a weight determination neural network according to the projection energy of each frequency point in the plurality of frequency points corresponding to each predetermined position in the plurality of predetermined positions and the energy information of the speech signal, comprising: Determine, by the control subnetwork, a signal-to-noise ratio characterization value of the speech signal according to energy information of the speech signal, and determine, according to the signal-to-noise ratio characterization value, whether a weight value of each frequency point in the multiple frequency points is a predetermined weight; If not, the calculation subnetwork determines a weight value of each frequency point in the multiple frequency points according to the projection energy of each frequency point in the multiple frequency points corresponding to each predetermined position in the multiple predetermined positions.
11. The method according to claim 10, It is characterized in that The control subnetwork extracts features from the energy information of the speech signal via a plurality of cascaded first feature extraction layers, and obtains a signal-to-noise ratio representation value according to the extracted features via a classification layer; The computing subnetwork extracts features of the projection energy of each frequency point among the multiple frequency points corresponding to each predetermined position among the multiple predetermined positions through a cascade of several second feature extraction layers, and obtains the weight of each frequency point among the multiple frequency points according to the extracted features through a linear regression layer.
12. The method according to claim 11, It is characterized in that The second feature extraction layer merges the extracted features with the features output by the corresponding first feature extraction layer in the control subnetwork, and outputs the merged features.
13. The method according to claim 3, It is characterized in that Determining the projection energy of the speech signal corresponding to each of the plurality of predetermined positions according to the weight values corresponding to each of the plurality of frequency points and the projection energy of each of the plurality of frequency points corresponding to each of the plurality of predetermined positions, comprises: performing weighted processing on the projection energy of each frequency point in the plurality of frequency points corresponding to each predetermined position in the plurality of predetermined positions according to a weight value corresponding to each frequency point in the plurality of frequency points; The weighted projection energies of each frequency point in the plurality of frequency points corresponding to each predetermined position in the plurality of predetermined positions are accumulated to obtain the projection energies of the speech signals corresponding to the plurality of predetermined positions.
14. The method according to claim 1, It is characterized in that Determining the position of the electronic device relative to the user during the voice call from a plurality of predetermined positions according to the position energy information includes: Acquire projection energy of a speech signal corresponding to each predetermined position of the plurality of predetermined positions; Identifying maximum position energy information from the projection energies of the speech signals corresponding to each of the plurality of predetermined positions; The location information of the electronic device is obtained from the multiple predetermined locations according to the maximum location energy information.
15. The method according to claim 1, It is characterized in that Also includes: Based on the position information of the electronic device relative to the user during the voice call, the voice signal is subjected to noise reduction processing to obtain a noise-reduced voice signal.
16. The method according to claim 1, It is characterized in that The position energy corresponds to each of a plurality of predetermined positions; Alternatively, determining the position of the electronic device relative to the user during the voice call from a plurality of predetermined positions may include: identifying one of the plurality of predetermined locations as a location of the electronic device relative to a user during a voice call; Or, determining the position of the electronic device relative to the user during the voice call from multiple predetermined positions based on the position energy information, including: obtaining multiple feature vectors corresponding to the multiple predetermined positions respectively based on the position energy information; and identifying one of the multiple predetermined positions as the position of the electronic device relative to the user during the voice call based on the feature vector corresponding to the user and the multiple feature vectors.
17. An electronic device, It is characterized in that It includes: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the method according to any one of claims 1-16.
18. A computer-readable storage medium, It is characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1-16.
