Azimuth-guided voice pickup method and voice pickup device

Abstract

The invention provides an orientation-guided voice pickup method and device, and the method comprises the steps: carrying out the short-time Fourier transform processing of an original voice signal in response to a voice attention instruction of a target user, and obtaining a binaural frequency domain feature which comprises the information of different orientations of an attended user; for any orientation information, Boolean direction coding processing is carried out on the orientation information to obtain discrete Boolean features; processing the plurality of discrete Boolean features by using a feature encoder to obtain a first linear feature under the target dimension; processing the binaural frequency domain features and the multiple pieces of azimuth information by using a spatial-temporal feature extraction network to obtain target convolution features under a target dimension; and processing the first linear feature and the target convolution feature by using a voice extraction network to obtain a target output voice highlighting the voice of the concerned user.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and more specifically, to a direction-guided speech pickup method, a speech pickup device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Target Speaker Extraction (TSE) is an important part of speech signal processing. Its core task is to separate and extract the voice of a specific target speaker from mixed audio containing multiple speakers. In practical applications, TSE, as a front-end processing module, plays an important role in many fields such as remote conferencing, hearing aids, and in-vehicle voice interaction. In addition, effectively extracting and enhancing the speech signal of a specific speaker is also extremely important for improving the performance of speech recognition and speech synthesis systems. Summary of the invention

[0003] In view of this, the present application provides a direction-guided voice pickup method, a voice pickup device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] One aspect of the present application provides a direction-guided voice pickup method, comprising:

[0005] In response to the target user's voice attention instruction, the original voice signal is short-time Fourier transform processed to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0006] For any of the above-mentioned azimuth information, the above-mentioned azimuth information is subjected to Boolean direction encoding processing to obtain a discrete Boolean feature;

[0007] Processing a plurality of the above discrete Boolean features using a feature encoder to obtain a first linear feature under a target dimension;

[0008] Processing the binaural frequency domain features and the plurality of the orientation information using a spatiotemporal feature extraction network to obtain target convolution features under the target dimension;

[0009] The first linear feature and the target convolution feature are processed by a speech extraction network to obtain a target output speech that highlights the voice of the user being focused.

[0010] Another aspect of the present application provides a voice pickup device with azimuth guidance, comprising:

[0011] A transformation module, configured to perform short-time Fourier transform processing on the original voice signal in response to the voice attention instruction of the target user to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0012] A first encoding module is used for performing Boolean direction encoding processing on any of the above-mentioned position information to obtain a discrete Boolean feature;

[0013] A second encoding module is used to process the plurality of discrete Boolean features using a feature encoder to obtain a first linear feature under a target dimension;

[0014] A feature extraction module, used to process the binaural frequency domain features and the plurality of the orientation information using a spatiotemporal feature extraction network to obtain a target convolution feature under the target dimension;

[0015] The voice pickup module is used to process the first linear feature and the target convolution feature using a voice extraction network to obtain a target output voice that highlights the voice of the user being concerned.

[0016] Another aspect of the present application provides an electronic device, comprising:

[0017] one or more processors;

[0018] a memory for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0020] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0021] Another aspect of the present application provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the method as described above when executed.

[0022] According to an embodiment of the present application, after the target user gives the direction of attention, the original voice signal is subjected to short-time Fourier transform processing to obtain binaural frequency domain features, and for any azimuth information of the focused user, the azimuth information is subjected to Boolean direction encoding processing to obtain discrete Boolean features, and then a feature encoder is used to process multiple discrete Boolean features to obtain the first linear feature under the target dimension, and a spatiotemporal feature extraction network is used to process the binaural frequency domain features and multiple azimuth information to obtain the target convolution feature under the target dimension, and finally a voice extraction network is used to process the first linear feature and the target convolution feature to obtain the target output voice that highlights the voice of the focused user. Because the voice pickup method of the present application clearly distinguishes the direction from which the voice of the focused user reaches the target user through Boolean direction encoding, the ability to extract the voice of the focused user is enhanced without relying on a specific microphone array configuration, thereby improving the clarity of the voice pickup of the focused user and the performance of the voice pickup system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0024] Figure 1 An exemplary system architecture to which a voice pickup method according to an embodiment of the present application can be applied is shown;

[0025] Figure 2 A flow chart of a voice pickup method according to an embodiment of the present application is shown;

[0026] Figure 3 A flow chart of a voice pickup method according to another embodiment of the present application is shown;

[0027] Figure 4 A comparison diagram of the spectrograms of the original speech signal, the frequency domain signal and the target output speech in the case of attention switching according to an embodiment of the present application is shown;

[0028] Figure 5 A schematic diagram showing the relationship between BiTSE SI-SNRi and the steering angle when the concerned user moves according to an embodiment of the present application;

[0029] Figure 6 A block diagram of a voice pickup device according to an embodiment of the present application is shown;

[0030] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application.

[0032] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0035] There are many types of clues to identify the target speaker, such as video information, registered audio, and the direction of arrival (DOA) of the sound. Video-based TSE technology often encounters the problem of audio and video being out of sync. Methods that rely on registered audio may encounter problems such as difficulty in obtaining registered audio or difficulty in matching due to changes in voiceprints over time. In contrast, using DOA information to identify speakers is more convenient. DOA information can be directly estimated through a multi-microphone array or measured with the help of a gyroscope or ultrasonic sensor. DOA-based TSE technology is more suitable for integration into binaural devices such as hearing aids or headphones, and is therefore gradually becoming a hot topic in the research field.

[0036] At present, many DOA-based spatial positioning technologies have shown excellent performance when using regularly arranged and evenly spaced microphone arrays. For example, in the related art, the cosine value of DOA is directly used as a spatial feature, and is combined with the mixed audio signal and processed by an activation function. However, this method based on coarse-grained DOA information may not be able to capture the detailed features in the mixed audio finely. On the other hand, another related technology proposes to align the multi-channel signal with the direction of the DOA of the target speaker by using beamforming technology, but the effect of this method is highly dependent on the specific layout of the microphone array. In general, the above-mentioned TSE methods all rely on regular microphone arrays, and there is a common problem of granularity misalignment between DOA features and mixed audio signals.

[0037] In binaural hearing systems, microphones are usually arranged in an irregular manner to form a small array (such as 2 to 4 microphones built into each hearing aid), or only one sensor is set on each side. Due to the difference in interaural distance between individuals and the occlusion effect of the head on sound, the performance of traditional DOA estimation technology based on regularly arranged microphone arrays may not meet expectations in binaural environments. This is because in binaural scenarios, the specific layout of microphones is often not precisely known, which makes it difficult for these methods to accurately capture directional information, thus affecting the effect of binaural TSE (BiTSE). In addition, in order to improve the performance of BiTSE, it is also necessary to overcome the granularity mismatch between DOA cues and mixed audio signals.

[0038] In view of this, an embodiment of the present application provides a direction-guided voice pickup method and a voice pickup device, the method comprising: responding to a voice attention instruction of a target user, performing short-time Fourier transform processing on an original voice signal to obtain binaural frequency domain features, wherein the binaural frequency domain features include different direction information of the focused user; for any direction information, performing Boolean direction encoding processing on the direction information to obtain discrete Boolean features; using a feature encoder to process multiple discrete Boolean features to obtain a first linear feature under a target dimension; using a spatiotemporal feature extraction network to process the binaural frequency domain features and multiple direction information to obtain a target convolution feature under a target dimension; using a voice extraction network to process the first linear feature and the target convolution feature to obtain a target output voice that highlights the voice of the focused user.

[0039] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0040] In the embodiments of the present application, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0041] Figure 1 An exemplary system architecture 100 to which the voice pickup method according to an embodiment of the present application can be applied is shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, in order to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0042] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0043] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0044] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0046] It should be noted that the voice pickup method provided in the embodiment of the present application can generally be performed by the server 105. Accordingly, the voice pickup device provided in the embodiment of the present application can generally be arranged in the server 105. The voice pickup method provided in the embodiment of the present application can also be performed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the voice pickup device provided in the embodiment of the present application can also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the voice pickup method provided in the embodiment of the present application can also be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or can also be performed by other terminal devices that are different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Correspondingly, the voice pickup device provided in the embodiment of the present application can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0048] Figure 2 A flow chart of a voice pickup method according to an embodiment of the present application is shown.

[0049] like Figure 2 As shown, the direction-guided voice pickup method includes operations S201 to S205.

[0050] In operation S201, in response to a target user's voice attention instruction, short-time Fourier transform processing is performed on the original voice signal to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0051] In operation S202, for any azimuth information, Boolean direction encoding is performed on the azimuth information to obtain a discrete Boolean feature;

[0052] In operation S203, a feature encoder is used to process the plurality of discrete Boolean features to obtain a first linear feature in a target dimension;

[0053] In operation S204, binaural frequency domain features and multiple orientation information are processed using a spatiotemporal feature extraction network to obtain target convolution features in a target dimension;

[0054] In operation S205, the first linear feature and the target convolutional feature are processed by using a speech extraction network to obtain a target output speech that highlights the voice of the focused user.

[0055] According to an embodiment of the present application, the voice attention instruction may be various types of sensors set on the target user, which can sense the orientation of the user's head, eyes (or other body parts, such as the direction of body rotation), and based on the direction detected by the sensor, the speaker corresponding to the orientation is determined from multiple speakers around the target user as the user to be paid attention to. At this time, an electronic device (such as a hearing aid) connected to the sensor can automatically generate the voice attention instruction.

[0056] It should be noted that the electronic device can output a target output voice of the voice of the concerned user, and the target output voice can filter or weaken the voices of other irrelevant speakers and enhance the voice of the concerned user. The electronic device can also convert the target output voice that highlights the voice of the concerned user into text, so that the hearing-impaired user can read the text to know the content of the speech of the concerned user.

[0057] According to the embodiment of the present application, each speaker has respective position information relative to the target user, and the position information can be represented by different position information, such as the speaker's position angle and elevation The following examples are based on the azimuth angle. and elevation Two types of position information are exemplified.

[0058] According to an embodiment of the present application, after the electronic device generates a voice attention instruction, the original voice signal containing the voice of the user being paid attention to is subjected to short-time Fourier transform processing to obtain binaural frequency domain features. , binaural frequency domain features It can be calculated by formula (1):

[0059] (1)

[0060] Among them, located at The speaker S is the focused user H, and the rest are located at of Speaker are considered as noise sources. and denote the head-related transfer functions from the focused user and other speakers to the left ear l and the right ear r, respectively.

[0061] According to an embodiment of the present application, Boolean Directional Embedding (BDE) refers to encoding the direction of arrival (DOA) clues of the sound in the context of target speaker extraction (TSE) to clearly distinguish the direction of the user being focused. and elevation Any orientation information in the , by performing Boolean direction encoding on the orientation information, a discrete Boolean feature can be obtained and .

[0062] According to an embodiment of the present application, the obtained discrete Boolean feature and Input into the feature encoder to get the target dimension The first linear feature under. Among them, Represents the clock unit associated with the original speech signal ,in, , t represents time, f is the frequency information, T is the entire duration of the original speech signal, and D is the quantitative dimension that represents the direction between the speaker and the target user.

[0063] According to an embodiment of the present application, a spatiotemporal feature extraction network is used to simultaneously process binaural frequency domain features. , azimuth and elevation , and obtain the target convolution feature under the target dimension. The obtained first linear feature and the target convolution feature are input into the speech extraction network to obtain the target output speech that highlights the voice of the user being concerned.

[0064] According to an embodiment of the present application, after the target user gives the focus direction, the original voice signal is subjected to short-time Fourier transform processing to obtain binaural frequency domain features, and for any azimuth information of the focused user, the azimuth information is subjected to Boolean direction coding processing to obtain discrete Boolean features, and then a feature encoder is used to process multiple discrete Boolean features to obtain the first linear feature under the target dimension, and a spatiotemporal feature extraction network is used to process the binaural frequency domain features and multiple azimuth information to obtain the target convolution feature under the target dimension, and finally a voice extraction network is used to process the first linear feature and the target convolution feature to obtain the target output voice that highlights the voice of the focused user. Because the voice pickup method of the present application clearly distinguishes the direction from which the voice of the focused user reaches the target user through Boolean direction coding, the ability to extract the voice of the focused user is enhanced without relying on a specific microphone array configuration, thereby improving the clarity of the voice pickup of the focused user.

[0065] According to an embodiment of the present application, the orientation information is subjected to Boolean direction encoding processing to obtain discrete Boolean features, including:

[0066] Perform vector mapping processing on the orientation information to obtain a multi-dimensional vector;

[0067] The multidimensional vectors are processed successively along the frequency direction and the time dimension direction to obtain discrete Boolean features.

[0068] According to an embodiment of the present application, in the process of Boolean direction encoding, the orientation information is first and Map them to a D-dimensional vector m respectively to get a multidimensional vector and , for example, for Multidimensional vector of As shown in formula (2):

[0069] (2)

[0070] Among them, when When the kth direction among D directions is matched, , the rest of the positions are zero, that is, m k represents the kth direction, and the T in the upper right corner is the transpose. Among them, D determines the spatial resolution.

[0071] According to the embodiment of the present application, in order to Alignment can be used to convert multidimensional vectors and Projecting into the time-frequency space, that is, projecting along the frequency direction and the time dimension direction, to obtain discrete Boolean features and .

[0072] According to an embodiment of the present application, the multidimensional vector is processed in sequence along the frequency direction and the time dimension direction to obtain discrete Boolean features, including:

[0073] Scaling the multidimensional vector along the frequency direction to obtain the frequency domain feature matrix;

[0074] The frequency domain feature matrix is ​​expanded along the time dimension to obtain discrete Boolean features.

[0075] According to an embodiment of the present application, the multidimensional vector and Scaling can expand the multidimensional vector into a frequency domain directional feature matrix , its form is shown in formula (3):

[0076] (3)

[0077] The matrix is ​​then further expanded along the time dimension.

[0078] According to an embodiment of the present application, after obtaining the frequency domain feature matrix and After that, it is expanded along the time dimension to obtain discrete Boolean features and .

[0079] According to an embodiment of the present application, the frequency domain feature matrix is ​​expanded along the time dimension to obtain discrete Boolean features, including:

[0080] In the case where the target user always pays attention to the followed user, a discrete Boolean feature is generated according to a plurality of frequency domain feature matrices;

[0081] In the case where the target user switches from paying attention to the focused user to paying attention to at least one switching user, based on the attention time of different target focused users, according to the frequency domain feature matrix of the target focused users at different attention times, a discrete Boolean feature is generated, wherein the target focused users include the focused user and at least one switching user, and different target focused users have different position information;

[0082] When the position information of the focused user changes, a discrete Boolean feature is generated according to a frequency domain feature matrix under different position information from the focused user.

[0083] According to an embodiment of the present application, when the target user always pays attention to the followed user and the followed user always remains stationary with the target user, a discrete Boolean feature is generated according to a plurality of frequency domain feature matrices. Specifically, a discrete Boolean feature can be generated according to T frequency domain feature matrices. and Generate discrete Boolean features, such as , the frequency domain feature matrix can be generated by formula (4) :

[0084] (4)

[0085] According to an embodiment of the present application, when the target user's attention is shifted from the azimuth of the followed users moved to the azimuth Switch user at time , , then the obtained frequency domain feature matrix It can be calculated by formula (5):

[0086] (5)

[0087] The above embodiment only shows one switching user. In fact, the attention of the target user can be switched to switching user 1, switching user 2, and so on in sequence along the time scale. , and its calculation formula is similar to formula (5).

[0088] According to the embodiment of the present application, when the concerned user is in a moving state during speaking, that is, when the azimuth information changes, the trajectory of the concerned user can be represented by a discrete azimuth set. Indicates that the azimuth Corresponding to the frequency domain feature matrix , t∈[1,2……T], then the discrete Boolean feature It can be calculated by formula (6):

[0089] (6)

[0090] Similarly, discrete Boolean features in different situations Calculation and The calculation of is similar.

[0091] Figure 3 A flow chart of a voice pickup method according to another embodiment of the present application is shown.

[0092] According to the embodiments of the present application, Figure 3 As shown, a feature encoder is used to process multiple discrete Boolean features to obtain a first linear feature in the target dimension, including: for any orientation information, a two-dimensional convolution is performed on the discrete Boolean feature corresponding to the orientation information to obtain an initial convolution feature; and a first linear layer is used to process multiple initial convolution features of different orientation information to obtain a first linear feature.

[0093] According to an embodiment of the present application, the main function of the first linear layer is to perform linear transformation, that is, to perform matrix multiplication on the input data and add a bias.

[0094] According to the embodiment of the present application, for the azimuth and elevation , respectively, using a two-dimensional (2D) convolution layer for processing to obtain two initial convolution features, and then using the first linear layer to process the two initial convolution features to obtain the first linear feature (D, D1).

[0095] According to an embodiment of the present application, binaural frequency domain features include left ear frequency domain features and right ear frequency domain features.

[0096] According to an embodiment of the present application, a spatiotemporal feature extraction network is used to process binaural frequency domain features and multiple orientation information to obtain target convolution features in the target dimension, including:

[0097] Based on the cosine direction function and the sine direction function, multiple azimuth information are processed respectively to generate target cosine distances and target sine distances corresponding to different azimuth information;

[0098] Perform channel-dimensional splicing processing on the left ear frequency domain features, the right ear frequency domain features, the target cosine distance, and the target sine distance to obtain splicing features;

[0099] Perform one-dimensional convolution on the concatenated features to obtain the target convolution features.

[0100] According to an embodiment of the present application, the azimuth is processed using a cosine direction function and elevation , get the target cosine distance At the same time, the sine direction function is used to process the azimuth and elevation , get the target sine distance .

[0101] According to an embodiment of the present application, the left ear frequency domain feature and right ear frequency domain characteristics , target cosine distance Sine distance to target Perform channel dimension splicing to obtain splicing features, and then perform one-dimensional (1D) convolution on the splicing features to obtain target convolution features, which have target dimensions .

[0102] According to an embodiment of the present application, multiple azimuth information is processed based on the cosine direction function and the sine direction function respectively to generate target cosine distances and target sine distances corresponding to different azimuth information, including:

[0103] For any direction function of the cosine direction function and the sine direction function, generating a time difference of arrival corresponding to different direction information according to the average interaural distance, the direction function and the direction information of the target user;

[0104] For any azimuth information, a theoretical phase difference is generated based on the arrival time difference and frequency information;

[0105] A target cosine distance or a target sine distance is generated according to the theoretical phase difference, the interaural phase difference and the direction function, wherein the interaural phase difference is generated based on binaural frequency domain features.

[0106] According to an embodiment of the present application, for a cosine directional function (CDF), the time difference of arrival (TDOA) of the focused user between the two ears of the target user can be calculated by formula (7):

[0107] (7)

[0108] in, is the speed of sound in air, Indicates the average interaural distance between the left and right ears of the target user.

[0109] According to an embodiment of the present application, the theoretical phase difference (TPD) caused by the focused user can be calculated by formula (8):

[0110] (8)

[0111] in, represents the sampling rate, and f represents the frequency information.

[0112] According to the embodiment of the present application, according to the theoretical phase difference , interaural phase difference (IPD) and cosine direction function to generate the target cosine distance C.

[0113] According to an embodiment of the present application, generating a target cosine distance or a target sine distance according to a theoretical phase difference, an interaural phase difference and a direction function includes:

[0114] In the case where the target user always pays attention to the focused user, an initial cosine distance or an initial sine distance is generated according to the theoretical phase difference, the interaural phase difference and the direction function, wherein the initial cosine distance and the initial sine distance represent the target cosine distance and the target sine distance respectively;

[0115] When the target user switches from paying attention to the focused user to paying attention to at least one switching user, generating a target cosine distance or a target sine distance according to initial cosine distances or initial sine distances with target focused users at different focusing times, wherein the target focused users include the focused user and at least one switching user, and different target focused users have different position information;

[0116] When the position information of the focused user changes, a target cosine distance or a target sine distance is generated according to an initial cosine distance or an initial sine distance under position information different from that of the focused user.

[0117] According to the embodiment of the present application, when the target user always pays attention to the concerned user, according to the theoretical phase difference , interaural phase difference IPD and cosine direction function to generate the initial cosine distance , the target cosine distance is calculated by formula (9):

[0118] (9)

[0119] According to an embodiment of the present application, when the target user switches from following the followed user to following at least one switching user, a target cosine distance is generated according to the initial cosine distance with the target following user at different following times t. , specifically, the target cosine distance is calculated by formula (10) Calculation:

[0120] (10)

[0121] According to the embodiment of the present application, when the position information of the concerned user changes, that is, the concerned user moves, the theoretical phase difference TPD is is a function of The target cosine distance can be expressed as , is the formula (9) splicing.

[0122] According to the embodiment of the present application, the sine directional function (SDF) supplements the features based on the cosine directional function. Similar to the calculations of formula (7) and formula (8), the arrival time difference and theoretical phase difference corresponding to the sine directional function can be obtained, thereby calculating the target sine distance .

[0123] According to an embodiment of the present application, when the target user always follows the followed user, the target sinusoidal distance is calculated based on formula (11):

[0124] (11)

[0125] According to an embodiment of the present application, when the followed user switches to the followed user or moves, the target sinusoidal distance R when the followed user switches to the followed user or moves can be calculated with reference to a calculation method based on a cosine direction function.

[0126] According to the embodiments of the present application, Figure 3 As shown, the first linear feature and the target convolution feature are processed by the speech extraction network to obtain the target output speech that highlights the focused user, including:

[0127] The first linear feature and the target convolution feature are processed based on the Hadamard product to obtain the spatiotemporal fusion feature;

[0128] The target learning network is used to process the spatiotemporal fusion features to obtain the target recognition features, wherein the target learning network includes L narrowband transformers connected in sequence;

[0129] Perform linear processing on the target recognition features to obtain frequency domain signals;

[0130] The frequency domain signal is inversely transformed to obtain the target output speech of the time domain type.

[0131] According to an embodiment of the present application, the narrowband converter is the NBC 2 block in the figure, and its English name is narrow-bandConformer, wherein the number of narrowband converters can be specifically set according to actual needs. Considering the voice pickup effect and the resources required for data processing, 4 narrowband converters can be set.

[0132] According to an embodiment of the present application, the Hadamard product is calculated on the first linear feature and the target convolution feature based on the target dimension to obtain the spatiotemporal fusion feature, and then the spatiotemporal fusion feature is input into the target learning network to obtain the target recognition feature, and then a linear layer is used to perform linear processing on the target recognition feature to obtain the frequency domain signal. The frequency domain signal of the left ear can be used Indicates that the frequency domain signal of the right ear can be expressed as Said, yes , Perform inverse transform processing to obtain the target output speech of the time domain type, that is, the target output speech of the left ear can be used Indicates that the target output speech for the left ear can be expressed as express.

[0133] According to an embodiment of the present application, in the training stage, the voice pickup model of the embodiment of the present application includes the above-mentioned one-dimensional convolutional layer, feature encoder, spatiotemporal feature extraction network and voice extraction network, and its training samples are the original training voice signals and the corresponding position information of the training user being focused. At the same time, the training process is labeled supervised training.

[0134] Specifically, the training is performed using the Adam optimizer, with the initial learning rate set to 0.001, and if the loss on the training set does not decrease within 3 epochs, it is halved. The scale-invariant signal-to-distortion ratio (SI-SDR) is used as the loss function, as shown in formula (12):

[0135] (12)

[0136] in, and They represent the clean signals of the focused training user in the left ear and the right ear of the target training user, respectively. and Represent the predicted training output speech at the left and right ears respectively. and Represents the loss components between the training output speech of the left ear and the right ear and the label signal, and their weights are equal.

[0137] Figure 4 A comparison diagram of the spectrum diagrams of the original speech signal, the frequency domain signal and the target output speech in the case of attention switching according to an embodiment of the present application is shown. Figure 5 A schematic diagram showing the relationship between BiTSE SI-SNRi and the steering angle when the focused user moves according to an embodiment of the present application.

[0138] According to the embodiments of the present application, in order to verify the effectiveness of the voice pickup method of the embodiments of the present application, the present application designs a comparative test between the voice pickup method and the existing TSE method in an environment with reverberation and noise, so as to further demonstrate the advantages and effectiveness of the method of the present invention.

[0139] The speech pickup method of the embodiment of the present application is trained and experimentally verified on the Intel Xeon E5-2680 CPU and NVIDIA A100 GPU. The model is built using Python language and PyTorch framework.

[0140] Static dataset: The static dataset uses the WSJ0-2mix dataset and randomly samples the head-related impulse response (HRIR) from the CIPIC HRTF dataset to construct 30-hour, 10-hour, and 5-hour training, validation, and test subsets, respectively. The sampling rate is The CIPIC HRTF dataset contains 45 subjects with 25 azimuths ( to ) and 50 elevation angles ( to ). In the training set, we used 27 subjects; for testing, we used 9 subjects that did not appear in training. In addition, two versions of the dataset were generated: one with reverberation and the other with noise. The former was synthesized using the BRIR Sim Set, with reverberation times ranging from 0.1 s to 1.0 s in training and {0.3, 0.6, 0.9} s in testing. For the latter, the added noise was from the DEMAND dataset, with signal-to-noise ratios ranging from -15 to 2.5 dB.

[0141] Speaker switching and moving datasets: These data are obtained based on static, reverberation-free and noise-free datasets. In the case of speaker switching, the switching time is randomly generated within the time range of the audio signal as a marker for speaker switching. In the case of moving speakers, consider that the speaker moves only in the horizontal plane (i.e. Change Every 1000 samples corresponds to an instantaneous The corresponding HRTF is convolved with the speech source to synthesize the audio signal of the moving speaker. The speaker moves continuously in one direction until the maximum angle ( ) and stops. The average moving speed of the speaker is / Second.

[0142] The voice pickup model proposed in this application includes a The dimensions of the NBC2 block BDE (discrete Boolean) features are initially set to , that is, the spatial resolution is After the feature encoder, Upsampled to , and the combined audio features are also sampled to 96 through a one-dimensional convolutional layer. The short-time Fourier transform uses a Hanning window with a window length of 32ms and a frame skip size of 16ms. The initial learning rate is set to 0.001. Set to 14.5cm.

[0143] In the experiments of this application, SI-SNR improvement (SI-SNRi), wideband perceptual speech quality evaluation (WB-PESQ) and narrowband PESQ (NB-PESQ) are used to measure the quality of the extracted speech of the target speaker (i.e., the user of interest).

[0144] Table 1 shows the performance of the speech pickup model at different angles, expressed as SI-SNRi (dB), WB-PESQ, and NB-PESQ.

[0145] Table 1. Comparison of different spatial orientation cues in the unaural and noiseless conditions.

[0146]

[0147] Table 1 shows the effect of different feature combinations on the performance of the BiTSE system in an undisturbed and noiseless environment. The top half includes spatiotemporal features (CDF, ​​SDF, IPD, and ΔSTFT), which were originally proposed for the TSE task with regular microphone arrays. The effect of these features is significantly reduced compared to related techniques due to the reduced number of available channels and unknown microphone spacing in binaural scenarios. CDF+IPD+ΔSTFT performs better in the case of small angular differences, but CDF+IPD+SDF performs better overall and is therefore used as the baseline in the following sections. From the bottom half of Table 1, using only the proposed BDE as a cue improves by 9.04 dB on SI-SNRi compared to the baseline, showing its effectiveness in BiTSE. This improvement is due to the fact that BDE contains more explicit directional information. Combining it with other spatiotemporal features further improves the performance (although the improvement is not significant), among which BDE+CDF+IPD+SDF (referred to as Proposed (Prop) in the following experiments) achieves the best results because spatiotemporal features can provide more fine-grained time-frequency features to complement the global cues captured by BDE.

[0148] Table 2 shows the performance comparison with related technologies under reverberation and noisy conditions. Under these two conditions, it can be clearly seen that using only the proposed BDE features can significantly outperform separation networks such as FaSNet-TAC and NBC2-small, as well as baseline systems based on traditional spatiotemporal features. This highlights the effectiveness of the proposed BDE in BiTSE. In addition, the speech pickup model proposed in this application combines the CDF+IPD+SDF features to further improve the performance to a certain extent.

[0149] Table 2. Comparison of different methods in the presence of noise and reverberation, performance index is SI-SNRi

[0150]

[0151] Speaker switching: Figure 4 An example of listener attention switching is shown. Two speakers are speaking, but the listener (i.e., target user) switches attention from speaker 1 (the focused user) to speaker 2 (the switching user) at 2.3 seconds. Although the speech sources have a large overlap across the frequency band, the proposed method still successfully removes the interfering speaker from the noisy measurement, allowing the listener to focus on the target more clearly. Figure 4 (a) is the original speech signal, Figure 4 (b) is the frequency domain signal, Figure 4 (c) Output speech for the target.

[0152] Speaker moves: Figure 5The invention shows that the voice pickup method proposed in the embodiment of the present invention can detect the moving target speaker at the turning angle. The target speaker gradually moves from the front right to the back right of the listener. In general, when the target speaker is in the front (e.g., ), the BDE-based method and the baseline system achieve the best BiTSE performance. The farther the target speaker is from the front direction, the lower the SI-SNRi. This is due to the head occlusion effect and the smaller binaural difference in the front direction. However, the curve also shows that the head occlusion effect does not show a monotonic change. Figure 5 The “Proposed” in the description refers to the voice pickup method of the embodiment of the present application.

[0153] Figure 6 A block diagram of a voice pickup device according to an embodiment of the present application is shown.

[0154] like Figure 6 As shown, the direction-guided voice pickup device 600 includes a transformation module 610, a first encoding module 620, a second encoding module 630, a feature extraction module 640, and a voice pickup module 650.

[0155] The transformation module 610 is used to perform short-time Fourier transform processing on the original voice signal in response to the voice attention instruction of the target user to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0156] The first encoding module 620 is used to perform Boolean direction encoding processing on any direction information to obtain a discrete Boolean feature;

[0157] A second encoding module 630 is used to process multiple discrete Boolean features using a feature encoder to obtain a first linear feature under a target dimension;

[0158] A feature extraction module 640 is used to process binaural frequency domain features and multiple orientation information using a spatiotemporal feature extraction network to obtain target convolution features in a target dimension;

[0159] The voice pickup module 650 is used to process the first linear feature and the target convolution feature using the voice extraction network to obtain a target output voice that highlights the voice of the user being focused.

[0160] According to an embodiment of the present application, after the target user gives the focus direction, the original voice signal is subjected to short-time Fourier transform processing to obtain binaural frequency domain features, and for any azimuth information of the focused user, the azimuth information is subjected to Boolean direction coding processing to obtain discrete Boolean features, and then a feature encoder is used to process multiple discrete Boolean features to obtain the first linear feature under the target dimension, and a spatiotemporal feature extraction network is used to process the binaural frequency domain features and multiple azimuth information to obtain the target convolution feature under the target dimension, and finally a voice extraction network is used to process the first linear feature and the target convolution feature to obtain the target output voice that highlights the voice of the focused user. Because the voice pickup method of the present application clearly distinguishes the direction from which the voice of the focused user reaches the target user through Boolean direction coding, the ability to extract the voice of the focused user is enhanced without relying on a specific microphone array configuration, thereby improving the clarity of the voice pickup of the focused user.

[0161] According to an embodiment of the present application, the first encoding module 620 includes:

[0162] A mapping submodule is used to perform vector mapping processing on the orientation information to obtain a multi-dimensional vector;

[0163] The time-frequency processing submodule is used to process the multidimensional vectors in sequence along the frequency direction and the time dimension direction to obtain discrete Boolean features.

[0164] According to an embodiment of the present application, the time-frequency processing submodule includes:

[0165] A scaling unit, used for scaling the multidimensional vector along the frequency direction to obtain a frequency domain feature matrix;

[0166] The expansion unit is used to expand the frequency domain feature matrix along the time dimension to obtain discrete Boolean features.

[0167] According to an embodiment of the present application, the expansion unit includes:

[0168] The first generating subunit is used to generate discrete Boolean features according to a plurality of frequency domain feature matrices when the target user always follows the followed user;

[0169] The second generating subunit is used for generating discrete Boolean features based on the attention time of different target attention users and the frequency domain feature matrix of the target attention users at different attention times when the target user switches from paying attention to the followed user to paying attention to at least one switching user, wherein the target attention users include the followed user and at least one switching user, and different target attention users have different position information;

[0170] The third generating subunit is used to generate discrete Boolean features according to the frequency domain feature matrix under different position information of the followed user when the position information of the followed user changes.

[0171] According to an embodiment of the present application, the second encoding module 630 includes:

[0172] A convolution unit is used for performing two-dimensional convolution processing on the discrete Boolean features corresponding to any orientation information to obtain initial convolution features;

[0173] The first linear unit is used to process multiple initial convolution features of different orientation information using the first linear layer to obtain a first linear feature.

[0174] According to an embodiment of the present application, binaural frequency domain features include left ear frequency domain features and right ear frequency domain features.

[0175] According to an embodiment of the present application, the feature extraction module 640 includes:

[0176] A direction processing submodule is used to process multiple azimuth information based on a cosine direction function and a sine direction function, respectively, to generate a target cosine distance and a target sine distance corresponding to different azimuth information;

[0177] A dimension splicing submodule is used to perform channel dimension splicing processing on the left ear frequency domain features and the right ear frequency domain features, the target cosine distance and the target sine distance to obtain a splicing feature;

[0178] The convolution submodule is used to perform one-dimensional convolution processing on the concatenated features to obtain the target convolution features.

[0179] According to an embodiment of the present application, the direction processing submodule includes:

[0180] A first generating unit is used to generate, for any direction function of the cosine direction function and the sine direction function, a time difference of arrival corresponding to different direction information according to the average interaural distance, the direction function and the direction information of the target user;

[0181] The second generating unit is used to generate a theoretical phase difference according to the arrival time difference and the frequency point information for any azimuth information;

[0182] The third generating unit is used to generate a target cosine distance or a target sine distance according to a theoretical phase difference, an interaural phase difference and a direction function, wherein the interaural phase difference is generated based on binaural frequency domain features.

[0183] According to an embodiment of the present application, the third generating unit includes:

[0184] A fourth generating subunit is used to generate an initial cosine distance or an initial sine distance according to the theoretical phase difference, the interaural phase difference and the direction function when the target user always pays attention to the focused user, wherein the initial cosine distance and the initial sine distance represent the target cosine distance and the target sine distance respectively;

[0185] a fifth generating subunit, configured to generate a target cosine distance or a target sine distance according to initial cosine distances or initial sine distances with target focusing users at different focusing times when the target user switches from focusing on the focused user to focusing on at least one switching user, wherein the target focusing users include the focused user and at least one switching user, and different target focusing users have different position information;

[0186] The sixth generating subunit is used to generate a target cosine distance or a target sine distance according to an initial cosine distance or an initial sine distance with different azimuth information from the followed user when the azimuth information of the followed user changes.

[0187] According to an embodiment of the present application, the voice pickup module 650 includes:

[0188] A spatiotemporal fusion unit, used for processing the first linear feature and the target convolution feature based on the Hadamard product to obtain a spatiotemporal fusion feature;

[0189] A deep learning unit, used for processing the spatiotemporal fusion features by using a target learning network to obtain target recognition features, wherein the target learning network includes a plurality of narrowband converters connected in sequence;

[0190] The second linear unit is used to perform linear processing on the target recognition feature to obtain a frequency domain signal;

[0191] The inverse transform unit is used to perform inverse transform processing on the frequency domain signal to obtain a target output speech of the time domain type.

[0192] According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0193] For example, any of the transformation module 610, the first encoding module 620, the second encoding module 630, the feature extraction module 640, and the voice pickup module 650 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present application, at least one of the transformation module 610, the first encoding module 620, the second encoding module 630, the feature extraction module 640, and the voice pickup module 650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the transformation module 610, the first encoding module 620, the second encoding module 630, the feature extraction module 640, and the voice pickup module 650 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be performed.

[0194] It should be noted that the voice pickup device part in the embodiment of the present application corresponds to the voice pickup method part in the embodiment of the present application. The description of the voice pickup device part specifically refers to the voice pickup method part, which will not be repeated here.

[0195] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0196] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0197] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present application by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present application by executing the program stored in the one or more memories.

[0198] According to an embodiment of the present application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0199] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0200] The present application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0201] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0202] For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0203] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0204] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present application are executed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0205] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0206] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages, and specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0207] The embodiments of the present application are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present application. Although each embodiment is described above, this does not mean that the measures in each embodiment cannot be used in combination advantageously. The present application does not depart from the scope of the present application, and those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A method for voice pickup with azimuth guidance, characterized in that: include: In response to the target user's voice attention instruction, the original voice signal is short-time Fourier transform processed to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user; For any of the azimuth information, performing Boolean direction encoding processing on the azimuth information to obtain discrete Boolean features; Processing the plurality of discrete Boolean features using a feature encoder to obtain a first linear feature under a target dimension; Processing the binaural frequency domain features and the plurality of the orientation information using a spatiotemporal feature extraction network to obtain a target convolution feature under the target dimension; The first linear feature and the target convolutional feature are processed by a speech extraction network to obtain a target output speech that highlights the voice of the focused user.

2. The method according to claim 1, characterized in that The orientation information is subjected to Boolean direction encoding processing to obtain discrete Boolean features, including: Performing vector mapping processing on the orientation information to obtain a multi-dimensional vector; The multidimensional vector is processed in sequence along the frequency direction and the time dimension direction to obtain the discrete Boolean feature.

3. The method according to claim 2, characterized in that The multidimensional vector is processed in sequence along the frequency direction and the time dimension direction to obtain the discrete Boolean feature, including: Scaling the multidimensional vector along the frequency direction to obtain a frequency domain feature matrix; The frequency domain feature matrix is ​​expanded along the time dimension to obtain the discrete Boolean feature.

4. The method according to claim 3, characterized in that The frequency domain feature matrix is ​​expanded along the time dimension to obtain the discrete Boolean feature, including: In the case where the target user always pays attention to the concerned user, generating the discrete Boolean feature according to a plurality of the frequency domain feature matrices; In the case where the target user switches from paying attention to the focused user to paying attention to at least one switching user, based on the attention time of different target focused users, the discrete Boolean feature is generated according to the frequency domain feature matrix of the target focused users at different attention times, wherein the target focused users include the focused user and the at least one switching user, and different target focused users have different position information; When the position information of the focused user changes, the discrete Boolean feature is generated according to a frequency domain feature matrix under position information different from that of the focused user.

5. The method according to claim 1, characterized in that The plurality of discrete Boolean features are processed by a feature encoder to obtain a first linear feature under a target dimension, including: For any of the position information, performing two-dimensional convolution processing on the discrete Boolean features corresponding to the position information to obtain initial convolution features; The first linear feature is obtained by processing the multiple initial convolution features of different orientation information using a first linear layer.

6. The method according to claim 1 or 5, characterized in that: The binaural frequency domain features include left ear frequency domain features and right ear frequency domain features; The binaural frequency domain features and the plurality of the position information are processed by a spatiotemporal feature extraction network to obtain the target convolution features under the target dimension, including: Based on the cosine direction function and the sine direction function, the plurality of azimuth information are processed respectively to generate target cosine distances and target sine distances corresponding to different azimuth information; Performing channel dimension splicing processing on the left ear frequency domain feature and the right ear frequency domain feature, the target cosine distance and the target sine distance to obtain a splicing feature; Perform one-dimensional convolution processing on the splicing features to obtain the target convolution features.

7. The method according to claim 6, characterized in that The plurality of azimuth information are processed respectively based on the cosine direction function and the sine direction function to generate target cosine distances and target sine distances corresponding to different azimuth information, including: For any direction function of the cosine direction function and the sine direction function, generating an arrival time difference corresponding to different direction information according to the average interaural distance of the target user, the direction function and the direction information; For any of the azimuth information, generating a theoretical phase difference according to the arrival time difference and the frequency point information; The target cosine distance or the target sine distance is generated according to the theoretical phase difference, the interaural phase difference and the direction function, wherein the interaural phase difference is generated based on the binaural frequency domain features.

8. The method according to claim 7, characterized in that Generating the target cosine distance or the target sine distance according to the theoretical phase difference, the interaural phase difference and the direction function, comprising: In the case where the target user always pays attention to the focused user, generating an initial cosine distance or an initial sine distance according to the theoretical phase difference, the interaural phase difference and the direction function, wherein the initial cosine distance and the initial sine distance represent the target cosine distance and the target sine distance respectively; In the case where the target user switches from paying attention to the focused user to paying attention to at least one switching user, generating the target cosine distance or the target sine distance according to the initial cosine distance or the initial sine distance with the target focused user at different focusing times, wherein the target focused users include the focused user and the at least one switching user, and different target focused users have different position information; When the position information of the focused user changes, the target cosine distance or the target sine distance is generated according to the initial cosine distance or the initial sine distance under the position information different from that of the focused user.

9. The method according to claim 1, characterized in that: Processing the first linear feature and the target convolutional feature using a speech extraction network to obtain a target output speech that highlights the focused user includes: Processing the first linear feature and the target convolution feature based on the Hadamard product to obtain a spatiotemporal fusion feature; Processing the spatiotemporal fusion features using a target learning network to obtain target recognition features, wherein the target learning network includes a plurality of narrowband converters connected in sequence; Performing linear processing on the target recognition feature to obtain a frequency domain signal; The frequency domain signal is inversely transformed to obtain the target output speech of the time domain type.

10. A voice pickup device for azimuth guidance, characterized in that: include: A transformation module, configured to perform short-time Fourier transform processing on the original voice signal in response to the voice attention instruction of the target user to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user; A first encoding module is used for performing Boolean direction encoding processing on any of the position information to obtain a discrete Boolean feature; A second encoding module, used for processing the plurality of discrete Boolean features using a feature encoder to obtain a first linear feature under a target dimension; A feature extraction module, used for processing the binaural frequency domain features and the plurality of the orientation information using a spatiotemporal feature extraction network to obtain a target convolution feature under the target dimension; The voice pickup module is used to process the first linear feature and the target convolution feature using a voice extraction network to obtain a target output voice that highlights the voice of the user being concerned.

Images