Voice pickup method and voice pickup device for direction guidance

Abstract

The present application provides a direction-guided voice pickup method and a voice pickup device, which include: responding to a target user's voice attention instruction, performing short-time Fourier transform processing on an original voice signal to obtain binaural frequency domain features, wherein the binaural frequency domain features include different direction information of the focused user; for any direction information, performing Boolean direction encoding processing on the direction information to obtain discrete Boolean features; using a feature encoder to process multiple discrete Boolean features to obtain a first linear feature in a target dimension; using a spatiotemporal feature extraction network to process the binaural frequency domain features and multiple direction information to obtain a target convolution feature in the target dimension; using a voice extraction network to process the first linear feature and the target convolution feature to obtain a target output voice that highlights the voice of the focused user.

Description

Technical Field

[0001] The present application relates to the technical field of voice signal processing, and more specifically, to a direction-guided voice pickup method, a voice pickup device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Target speaker extraction (TSE) is a crucial step in speech signal processing. Its core task is to separate and extract the voice of a specific target speaker from mixed audio containing multiple speakers. In practical applications, TSE, as a front-end processing module, plays a vital role in a variety of fields, including teleconferencing, hearing aids, and in-vehicle voice interaction. Furthermore, effectively extracting and enhancing the speech signal of a specific speaker is crucial for improving the performance of speech recognition and speech synthesis systems. Summary of the Invention

[0003] In view of this, the present application provides a direction-guided voice pickup method, a voice pickup device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] One aspect of the present application provides a direction-guided voice pickup method, comprising:

[0005] In response to the target user's voice attention instruction, the original voice signal is subjected to short-time Fourier transform processing to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0006] For any of the above-mentioned direction information, performing Boolean direction encoding processing on the above-mentioned direction information to obtain discrete Boolean features;

[0007] Processing the plurality of the above discrete Boolean features using a feature encoder to obtain a first linear feature under the target dimension;

[0008] Processing the binaural frequency domain features and the plurality of orientation information using a spatiotemporal feature extraction network to obtain target convolution features under the target dimension;

[0009] The first linear feature and the target convolution feature are processed by a speech extraction network to obtain a target output speech that highlights the voice of the user being focused.

[0010] Another aspect of the present application provides a direction-guided voice pickup device, comprising:

[0011] a transform module, configured to perform short-time Fourier transform processing on the original voice signal in response to the target user's voice attention instruction to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0012] A first encoding module is configured to perform Boolean direction encoding processing on any of the above-mentioned direction information to obtain a discrete Boolean feature;

[0013] A second encoding module is used to process the plurality of discrete Boolean features using a feature encoder to obtain a first linear feature under a target dimension;

[0014] A feature extraction module is used to process the binaural frequency domain features and the plurality of orientation information using a spatiotemporal feature extraction network to obtain a target convolution feature under the target dimension;

[0015] The voice pickup module is used to process the first linear feature and the target convolution feature using a voice extraction network to obtain a target output voice that highlights the voice of the user being focused.

[0016] Another aspect of the present application provides an electronic device, comprising:

[0017] one or more processors;

[0018] a memory for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0020] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0021] Another aspect of the present application provides a computer program product, which includes computer-executable instructions. When the instructions are executed, the instructions are used to implement the method described above.

[0022] According to an embodiment of the present application, after the target user gives the direction of attention, the original voice signal is subjected to short-time Fourier transform processing to obtain binaural frequency domain features. For any azimuth information of the focused user, the azimuth information is subjected to Boolean direction encoding processing to obtain discrete Boolean features. Then, a feature encoder is used to process multiple discrete Boolean features to obtain the first linear feature under the target dimension. At the same time, a spatiotemporal feature extraction network is used to process the binaural frequency domain features and multiple azimuth information to obtain the target convolution feature under the target dimension. Finally, a voice extraction network is used to process the first linear feature and the target convolution feature to obtain the target output voice that highlights the voice of the focused user. Because the voice pickup method of the present application clearly distinguishes the direction from which the voice of the focused user reaches the target user through Boolean direction encoding, the ability to extract the voice of the focused user is enhanced without relying on a specific microphone array configuration, thereby improving the clarity of the voice pickup of the focused user and the performance of the voice pickup system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0024] Figure 1 An exemplary system architecture to which the voice pickup method according to an embodiment of the present application can be applied is shown;

[0025] Figure 2 A flow chart of a voice pickup method according to an embodiment of the present application is shown;

[0026] Figure 3 A flow chart of a voice pickup method according to another embodiment of the present application is shown;

[0027] Figure 4 A comparison diagram of the spectrograms of the original speech signal, the frequency domain signal, and the target output speech in the case of attention switching according to an embodiment of the present application is shown;

[0028] Figure 5 A schematic diagram showing the relationship between BiTSE SI-SNRi and steering angle when the user of interest moves according to an embodiment of the present application;

[0029] Figure 6 A block diagram of a voice pickup device according to an embodiment of the present application is shown;

[0030] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0032] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0035] There are many types of clues for identifying the target speaker, such as video information, registered audio, and the direction of arrival (DOA). Video-based TSE technologies often encounter audio and video synchronization issues. Methods that rely on registered audio may encounter difficulties in obtaining registered audio or matching due to voiceprint changes over time. In contrast, using DOA information to identify speakers is more convenient. DOA information can be directly estimated using a multi-microphone array or measured using a gyroscope or ultrasonic sensor. DOA-based TSE technology is more suitable for integration into binaural devices such as hearing aids or headphones, and is therefore becoming a hot research topic.

[0036] At present, many DOA-based spatial positioning technologies have demonstrated excellent performance when using regularly arranged and evenly spaced microphone arrays. For example, in related technologies, the cosine value of DOA is directly used as a spatial feature, and is combined with the mixed audio signal and processed by an activation function. However, this method based on coarse-grained DOA information may not be able to capture the detailed features in the mixed audio in a fine manner. On the other hand, another related technology proposes to align the multi-channel signal with the direction of the DOA of the target speaker by using beamforming technology, but the effectiveness of this method is highly dependent on the specific layout of the microphone array. In general, the above-mentioned TSE methods all rely on regular microphone arrays and generally have the problem of granularity misalignment between DOA features and mixed audio signals.

[0037] In binaural hearing systems, microphones are typically arranged irregularly, forming small arrays (e.g., two to four microphones per hearing aid), or with only one sensor per side. Due to differences in interaural distance between individuals and the occlusion of sound by the head, traditional DOA estimation techniques based on regularly arranged microphone arrays may not perform as expected in binaural environments. This is because the specific microphone layout in binaural scenarios is often not precisely known, making it difficult for these methods to accurately capture directional information, thus affecting the effectiveness of binaural TSE (BiTSE). Furthermore, to improve BiTSE performance, it is necessary to overcome the granularity mismatch between DOA cues and the mixed audio signal.

[0038] In view of this, an embodiment of the present application provides a direction-guided voice pickup method and a voice pickup device, which includes responding to the voice attention instruction of the target user, performing short-time Fourier transform processing on the original voice signal to obtain binaural frequency domain features, wherein the binaural frequency domain features include different direction information of the focused user; for any direction information, performing Boolean direction encoding processing on the direction information to obtain discrete Boolean features; using a feature encoder to process multiple discrete Boolean features to obtain a first linear feature under the target dimension; using a spatiotemporal feature extraction network to process the binaural frequency domain features and multiple direction information to obtain a target convolution feature under the target dimension; using a voice extraction network to process the first linear feature and the target convolution feature to obtain a target output voice that highlights the voice of the focused user.

[0039] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and maintain the security of user personal information and network security.

[0040] In the embodiments of the present application, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0041] Figure 1 FIG. 1 shows an exemplary system architecture 100 to which the voice pickup method according to an embodiment of the present application can be applied. It should be noted that, Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0042] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0043] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0044] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0046] It should be noted that the voice pickup method provided in the embodiment of the present application can generally be performed by the server 105. Accordingly, the voice pickup device provided in the embodiment of the present application can generally be arranged in the server 105. The voice pickup method provided in the embodiment of the present application can also be performed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the voice pickup device provided in the embodiment of the present application can also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the voice pickup method provided in the embodiment of the present application can also be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or can also be performed by other terminal devices that are different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the voice pickup device provided in the embodiment of the present application can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0048] Figure 2 A flow chart of a voice pickup method according to an embodiment of the present application is shown.

[0049] like Figure 2 As shown, the direction-guided voice pickup method includes operations S201 to S205.

[0050] In operation S201, in response to a target user's voice attention instruction, a short-time Fourier transform is performed on an original voice signal to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0051] In operation S202, for any direction information, Boolean direction encoding processing is performed on the direction information to obtain discrete Boolean features;

[0052] In operation S203, a feature encoder is used to process the plurality of discrete Boolean features to obtain a first linear feature in the target dimension;

[0053] In operation S204, the binaural frequency domain features and the multiple orientation information are processed using a spatiotemporal feature extraction network to obtain target convolution features in the target dimension.

[0054] In operation S205 , the first linear feature and the target convolution feature are processed using a speech extraction network to obtain a target output speech that highlights the voice of the user being focused.

[0055] According to an embodiment of the present application, the voice attention instruction can be various types of sensors set on the target user, which can sense the orientation of the user's head, eyes (or other body parts, such as the direction of body rotation). Based on the direction detected by the sensor, the speaker corresponding to the orientation is determined from multiple speakers around the target user as the user to be paid attention to. At this time, the electronic device connected to the sensor (such as a hearing aid) can automatically generate the voice attention instruction.

[0056] It should be noted that the electronic device can output a target output voice for the voice of the focused user. This target output voice can filter out or weaken the voices of other irrelevant speakers while enhancing the voice of the focused user. The electronic device can also convert the target output voice that emphasizes the voice of the focused user into text, so that the hearing-impaired user can understand the content of the focused user's speech by reading the text.

[0057] According to the embodiment of the present application, each speaker has its own position information relative to the target user, and the position information can be represented by different position information, such as the speaker's position angle and elevation The following examples are based on the azimuth angle and elevation Two types of position information are exemplified.

[0058] According to an embodiment of the present application, after the electronic device generates a voice attention instruction, the original voice signal containing the voice of the user being paid attention to is subjected to short-time Fourier transform processing to obtain binaural frequency domain features. , binaural frequency domain features It can be calculated by formula (1):

[0059] (1)

[0060] Among them, located at The speaker S is the focused user H, and the rest are located at of Speaker is considered a noise source. and denote the head-related transfer functions from the focused user and other speakers to the left ear l and right ear r, respectively.

[0061] According to an embodiment of the present application, Boolean Directional Embedding (BDE) refers to encoding the direction of arrival (DOA) clues of the sound in the context of target speaker extraction (TSE) to clearly distinguish the direction of the user being focused. and elevation Any orientation information in the , the orientation information is processed by Boolean direction encoding, and the discrete Boolean feature can be obtained and .

[0062] According to an embodiment of the present application, the obtained discrete Boolean features and Input into the feature encoder to get the target dimension The first linear characteristic under . Among them, Represents the clock unit associated with the original speech signal ,in, , t represents time, f is the frequency information, T is the entire duration of the original speech signal, and D is the quantitative dimension that represents the direction between the speaker and the target user.

[0063] According to the embodiment of the present application, a spatiotemporal feature extraction network is used to simultaneously process binaural frequency domain features. , azimuth and elevation , obtaining the target convolution feature in the target dimension. The obtained first linear feature and the target convolution feature are input into the speech extraction network to obtain the target output speech that highlights the voice of the user being focused.

[0064] According to an embodiment of the present application, after the target user gives the direction of attention, the original voice signal is subjected to short-time Fourier transform processing to obtain binaural frequency domain features. For any azimuth information of the focused user, the azimuth information is subjected to Boolean direction encoding processing to obtain discrete Boolean features. Then, a feature encoder is used to process multiple discrete Boolean features to obtain the first linear feature under the target dimension. At the same time, a spatiotemporal feature extraction network is used to process the binaural frequency domain features and multiple azimuth information to obtain the target convolution feature under the target dimension. Finally, a voice extraction network is used to process the first linear feature and the target convolution feature to obtain the target output voice that highlights the voice of the focused user. Because the voice pickup method of the present application clearly distinguishes the direction from which the voice of the focused user reaches the target user through Boolean direction encoding, the ability to extract the voice of the focused user is enhanced without relying on a specific microphone array configuration, thereby improving the clarity of the voice pickup of the focused user.

[0065] According to an embodiment of the present application, Boolean direction encoding is performed on the orientation information to obtain discrete Boolean features, including:

[0066] Perform vector mapping processing on the orientation information to obtain a multi-dimensional vector;

[0067] The multidimensional vectors are processed sequentially along the frequency direction and the time dimension to obtain discrete Boolean features.

[0068] According to the embodiment of the present application, in the process of Boolean direction encoding, the orientation information is first and Map them into a D-dimensional vector m respectively to get a multidimensional vector and , for example, for Multidimensional vector of As shown in formula (2):

[0069] (2)

[0070] Among them, when When the kth direction among the D directions is matched, , the rest of the positions are zero, that is, m k represents the kth direction, and the T in the upper right corner is the transpose. Among them, D determines the spatial resolution.

[0071] According to the embodiment of the present application, in order to Alignment can be used to convert multidimensional vectors and Projecting into the time-frequency space, that is, projecting along the frequency direction and the time dimension direction, thus obtaining discrete Boolean features and .

[0072] According to an embodiment of the present application, the multidimensional vector is processed sequentially along the frequency direction and the time dimension direction to obtain discrete Boolean features, including:

[0073] Scaling the multidimensional vector along the frequency direction to obtain the frequency domain feature matrix;

[0074] The frequency domain feature matrix is ​​expanded along the time dimension to obtain discrete Boolean features.

[0075] According to an embodiment of the present application, the multidimensional vector and Scaling can expand the multidimensional vector into a frequency domain directional feature matrix , its form is shown in formula (3):

[0076] (3)

[0077] The matrix is ​​then further expanded along the time dimension.

[0078] According to the embodiment of the present application, after obtaining the frequency domain feature matrix and Afterwards, it is expanded along the time dimension to obtain discrete Boolean features and .

[0079] According to an embodiment of the present application, the frequency domain feature matrix is ​​expanded along the time dimension to obtain discrete Boolean features, including:

[0080] In the case where the target user always follows the followed user, discrete Boolean features are generated according to multiple frequency domain feature matrices;

[0081] When the target user switches from paying attention to the focused user to paying attention to at least one switching user, based on the attention times of different target focusing users, a discrete Boolean feature is generated according to the frequency domain feature matrix of the target focusing users at different attention times, wherein the target focusing users include the focused user and at least one switching user, and different target focusing users have different location information;

[0082] When the position information of the focused user changes, discrete Boolean features are generated according to the frequency domain feature matrix under the position information different from that of the focused user.

[0083] According to an embodiment of the present application, when the target user always pays attention to the followed user and the followed user always remains stationary with the target user, a discrete Boolean feature is generated based on multiple frequency domain feature matrices. Specifically, the discrete Boolean feature can be generated based on T frequency domain feature matrices. and Generate discrete Boolean features, such as , the frequency domain feature matrix can be generated by formula (4) :

[0084] (4)

[0085] According to an embodiment of the present application, when the target user's attention is shifted from the azimuth The users being followed are transferred to the azimuth Switch user at time , , then the obtained frequency domain feature matrix It can be calculated by formula (5):

[0086] (5)

[0087] The above embodiment only shows one switching user. In fact, the target user's attention can be switched to switching user 1, switching user 2, and so on in sequence along the time scale. , and its calculation formula is similar to formula (5).

[0088] According to the embodiment of the present application, when the user being followed is in a moving state during the process of speaking, that is, when the position information changes, the trajectory of the user being followed can be represented by a discrete set of position angles. Indicates that the azimuth Corresponding to the frequency domain feature matrix , t∈[1,2……T], then the discrete Boolean feature It can be calculated by formula (6):

[0089] (6)

[0090] Similarly, discrete Boolean features in different situations Calculation and The calculation is similar.

[0091] Figure 3 A flow chart of a voice pickup method according to another embodiment of the present application is shown.

[0092] According to the embodiments of the present application, Figure 3 As shown, a feature encoder is used to process multiple discrete Boolean features to obtain a first linear feature under the target dimension, including: for any orientation information, a two-dimensional convolution processing is performed on the discrete Boolean feature corresponding to the orientation information to obtain an initial convolution feature; and a first linear layer is used to process multiple initial convolution features of different orientation information to obtain a first linear feature.

[0093] According to an embodiment of the present application, the main function of the first linear layer is to perform linear transformation, that is, to perform matrix multiplication on the input data and add a bias.

[0094] According to the embodiment of the present application, for the azimuth and elevation , respectively, using a two-dimensional (2D) convolution layer for processing to obtain two initial convolution features, and then using the first linear layer to process the two initial convolution features to obtain the first linear feature (D, D1).

[0095] According to an embodiment of the present application, the binaural frequency domain features include left ear frequency domain features and right ear frequency domain features.

[0096] According to an embodiment of the present application, a spatiotemporal feature extraction network is used to process binaural frequency domain features and multiple orientation information to obtain target convolution features in the target dimension, including:

[0097] Based on the cosine direction function and the sine direction function, multiple azimuth information are processed respectively to generate target cosine distance and target sine distance corresponding to different azimuth information;

[0098] Perform channel-dimensional splicing processing on the left ear frequency domain features, the right ear frequency domain features, the target cosine distance, and the target sine distance to obtain splicing features;

[0099] Perform one-dimensional convolution on the spliced ​​features to obtain the target convolution features.

[0100] According to an embodiment of the present application, the azimuth angle is processed using the cosine direction function. and elevation , get the target cosine distance At the same time, the sine direction function is used to process the azimuth and elevation , get the target sine distance .

[0101] According to the embodiment of the present application, the left ear frequency domain feature and right ear frequency domain characteristics , target cosine distance Sine distance to target Perform channel dimension splicing to obtain splicing features, and then perform one-dimensional (1D) convolution on the splicing features to obtain target convolution features, which have target dimensions. .

[0102] According to an embodiment of the present application, multiple azimuth information is processed based on the cosine direction function and the sine direction function respectively to generate target cosine distances and target sine distances corresponding to different azimuth information, including:

[0103] For any of the cosine direction function and the sine direction function, generate arrival time differences corresponding to different direction information according to the average interaural distance, direction function, and direction information of the target user;

[0104] For any azimuth information, the theoretical phase difference is generated based on the arrival time difference and frequency information;

[0105] A target cosine distance or a target sine distance is generated according to the theoretical phase difference, the interaural phase difference and the direction function, wherein the interaural phase difference is generated based on binaural frequency domain features.

[0106] According to an embodiment of the present application, for the cosine directional function (CDF), the time difference of arrival (TDOA) of the focused user between the two ears of the target user can be calculated by formula (7):

[0107] (7)

[0108] in, is the speed of sound in air, Indicates the average interaural distance between the left and right ears of the target user.

[0109] According to an embodiment of the present application, the theoretical phase difference (TPD) caused by the focused user can be calculated by formula (8):

[0110] (8)

[0111] in, represents the sampling rate, and f represents the frequency information.

[0112] According to the embodiment of the present application, according to the theoretical phase difference , Interaural Phase Difference (IPD) and cosine direction function to generate the target cosine distance C.

[0113] According to an embodiment of the present application, generating a target cosine distance or a target sine distance based on the theoretical phase difference, the interaural phase difference, and the direction function includes:

[0114] In the case where the target user always pays attention to the focused user, an initial cosine distance or an initial sine distance is generated according to the theoretical phase difference, the interaural phase difference and the direction function, wherein the initial cosine distance and the initial sine distance represent the target cosine distance and the target sine distance respectively;

[0115] When the target user switches from following the followed user to following at least one switching user, generating a target cosine distance or a target sine distance based on initial cosine distances or initial sine distances with the target following users at different following times, wherein the target following users include the followed user and the at least one switching user, and different target following users have different location information;

[0116] When the position information of the focused user changes, a target cosine distance or a target sine distance is generated according to the initial cosine distance or the initial sine distance under the position information different from that of the focused user.

[0117] According to the embodiment of the present application, when the target user always pays attention to the concerned user, according to the theoretical phase difference , interaural phase difference IPD and cosine direction function to generate the initial cosine distance , specifically, the target cosine distance is calculated using formula (9):

[0118] (9)

[0119] According to an embodiment of the present application, when the target user switches from following the followed user to following at least one switching user, the target cosine distance is generated according to the initial cosine distance with the target following user at different following times t. , specifically the target cosine distance is calculated using formula (10) Calculation:

[0120] (10)

[0121] According to an embodiment of the present application, when the position information of the concerned user changes, that is, when the concerned user moves, the theoretical phase difference TPD is function, where the trajectory is given by The target cosine distance can be expressed as , is the formula (9) splicing.

[0122] According to the embodiment of the present application, the sine directional function (SDF) supplements the features based on the cosine directional function. Similar to the calculation of formula (7) and formula (8), the arrival time difference and theoretical phase difference corresponding to the sine directional function can be obtained, thereby calculating the target sinusoidal distance .

[0123] According to an embodiment of the present application, when the target user always follows the followed user, the target sine distance is calculated based on formula (11):

[0124] (11)

[0125] According to an embodiment of the present application, when the followed user switches to the followed user or moves, the target sine distance R when the followed user switches to the followed user or moves can be calculated with reference to a calculation method based on a cosine direction function.

[0126] According to the embodiments of the present application, Figure 3 As shown, the speech extraction network is used to process the first linear feature and the target convolution feature to obtain the target output speech that highlights the focused user, including:

[0127] The first linear feature and the target convolution feature are processed based on the Hadamard product to obtain the spatiotemporal fusion feature;

[0128] A target learning network is used to process the spatiotemporal fusion features to obtain target recognition features, wherein the target learning network includes L narrowband transformers connected in sequence;

[0129] Perform linear processing on the target recognition features to obtain frequency domain signals;

[0130] Perform inverse transform on the frequency domain signal to obtain the target output speech of the time domain type.

[0131] According to an embodiment of the present application, the narrowband converter is the NBC 2 block in the figure, and its English name is narrow-bandConformer. The number of narrowband converters can be set according to actual needs. Considering the voice pickup effect and the resources required for data processing, 4 narrowband converters can be set.

[0132] According to the embodiment of the present application, the Hadamard product calculation is performed on the first linear feature and the target convolution feature based on the target dimension to obtain the spatiotemporal fusion feature, and then the spatiotemporal fusion feature is input into the target learning network to obtain the target recognition feature, and then a linear layer is used to linearly process the target recognition feature to obtain the frequency domain signal. The frequency domain signal of the left ear can be used Indicates that the frequency domain signal of the right ear can be expressed as Said, yes 、 Perform inverse transform processing to obtain the target output speech of the time domain type, that is, the target output speech of the left ear can be used Indicates that the target output speech for the left ear can be expressed as express.

[0133] According to an embodiment of the present application, during the training stage, the voice pickup model of the embodiment of the present application includes the above-mentioned one-dimensional convolutional layer, feature encoder, spatiotemporal feature extraction network and voice extraction network, and its training samples are the original training voice signals and the corresponding orientation information of the training user being focused. At the same time, the training process is labeled supervised training.

[0134] Specifically, the training is performed using the Adam optimizer, with the initial learning rate set to 0.001. If the loss on the training set does not decrease within 3 epochs, it is halved. The scale-invariant signal-to-distortion ratio (SI-SDR) is used as the loss function, as shown in formula (12):

[0135] (12)

[0136] in, and They represent the clean signals of the focused training user in the left ear and right ear of the target training user, and Represent the predicted training output speech at the left and right ears respectively. and Represents the loss components between the training output speech of the left ear and the right ear and the label signal, and their weights are equal.

[0137] Figure 4 A comparison diagram of the spectrum of the original speech signal, the frequency domain signal and the target output speech in the case of attention switching according to an embodiment of the present application is shown. Figure 5 A schematic diagram showing the relationship between BiTSE SI-SNRi and the steering angle when the focused user moves according to an embodiment of the present application is shown.

[0138] According to the embodiments of the present application, in order to verify the effectiveness of the voice pickup method of the embodiments of the present application, the present application designed a comparative test of the voice pickup method with the existing TSE method in an environment with reverberation and noise, so as to further demonstrate the advantages and effectiveness of the method of the present invention.

[0139] The speech pickup method of the present embodiment was trained and experimentally verified on an Intel Xeon E5-2680 CPU and an NVIDIA A100 GPU. The model was built using Python and the PyTorch framework.

[0140] Static dataset: The static dataset uses the WSJ0-2mix dataset and randomly samples the head-related impulse response (HRIR) from the CIPIC HRTF dataset to construct 30 hours, 10 hours and 5 hours of training, validation and test subsets respectively. The sampling rate is The CIPIC HRTF dataset contains 45 subjects with 25 azimuth angles ( to ) and 50 elevation angles ( to ). We used 27 subjects in the training set; for testing, we used 9 subjects not included in training. Furthermore, we generated two versions of the dataset: one with reverberation and one with noise. The former was synthesized using the BRIR Sim Set, with reverberation times ranging from 0.1 to 1.0 seconds during training and {0.3, 0.6, 0.9} seconds for testing. For the latter, the added noise was from the DEMAND dataset, with signal-to-noise ratios ranging from -15 to 2.5 dB.

[0141] Speaker switching and movement datasets: These data are obtained based on static, reverberation-free, and noise-free datasets. In the case of speaker switching, the switching moment is randomly generated within the time range of the audio signal to serve as a marker for speaker switching. In the case of moving speakers, the speaker is considered to move only in the horizontal plane (i.e. Change Every 1000 samples corresponds to an instantaneous The corresponding HRTF is convolved with the speech source to synthesize the audio signal of the moving speaker. The speaker moves continuously in one direction until the maximum angle ( ) and stops. The average moving speed of the speaker is / Second.

[0142] The voice pickup model proposed in this application includes the NBC2-small The dimensions of the NBC2 block BDE (discrete Boolean) features are initially set to , that is, the spatial resolution is After the feature encoder, Upsampled to , and the combined audio features are also downsampled to 96 through a one-dimensional convolutional layer. The short-time Fourier transform uses a Hanning window with a window length of 32ms and a frame skip size of 16ms. The initial learning rate is set to 0.001. Set to 14.5cm.

[0143] In the experiments of this application, SI-SNR improvement (SI-SNRi), wideband perceptual speech quality evaluation (WB-PESQ), and narrowband PESQ (NB-PESQ) are used to measure the quality of the extracted speech of the target speaker (i.e., the user of interest).

[0144] Table 1 shows the performance of the voice pickup model at different angles, expressed as SI-SNRi (dB), WB-PESQ, and NB-PESQ.

[0145] Table 1. Comparison of different spatial direction cues under undisturbed and noiseless conditions.

[0146]

[0147] Table 1 shows the impact of different feature combinations on the performance of the BiTSE system in audible and noiseless environments. The top panel includes spatiotemporal features (CDF, ​​SDF, IPD, and ΔSTFT), which were originally proposed for TSE tasks with regular microphone arrays. Due to the reduced number of available channels and unknown microphone spacing in binaural scenarios, the effectiveness of these features is significantly reduced compared to related techniques. CDF+IPD+ΔSTFT performs better in cases with small angular disparity, but CDF+IPD+SDF offers better overall performance and is therefore used as the baseline in the following sections. The bottom panel of Table 1 shows that using only the proposed BDE as a cue improves SI-SNRi by 9.04 dB compared to the baseline, demonstrating its effectiveness in BiTSE. This improvement is due to the BDE's inclusion of more explicit directional information. Combining it with other spatiotemporal features further improves the performance (although the improvement is not significant), among which BDE+CDF+IPD+SDF (referred to as Proposed (Prop) in the following experiments) achieves the best results because spatiotemporal features can provide more fine-grained time-frequency features to complement the global cues captured by BDE.

[0148] Table 2 shows a performance comparison with related techniques under reverberant and noisy conditions. Under both conditions, it is clear that the proposed BDE features alone significantly outperform separation networks such as FaSNet-TAC and NBC2-small, as well as baseline systems based on traditional spatiotemporal features. This highlights the effectiveness of the proposed BDE in BiTSE. Furthermore, the proposed speech pickup model, combined with CDF+IPD+SDF features, further improves performance.

[0149] Table 2. Comparison of different methods in the presence of noise and reverberation, performance index is SI-SNRi

[0150]

[0151] Speaker switching: Figure 4 An example of listener attention switching is shown. Two speakers are speaking, but the listener (i.e., the target user) switches their attention from Speaker 1 (the focused user) to Speaker 2 (the switching user) at 2.3 seconds. Despite significant overlap of speech sources across the entire frequency band, the proposed method successfully removes the interfering speaker from the noisy measurement, allowing the listener to more clearly focus on the target. Figure 4 (a) is the original speech signal, Figure 4 (b) is the frequency domain signal, Figure 4 (c) Output speech for the target.

[0152] Speaker moves: Figure 5The speech pickup method proposed in the embodiment of the present application is shown in FIG. The target speaker gradually moves from the front right to the back right of the listener. In general, when the target speaker is in front (e.g., ), the BDE-based method and the baseline system achieve the best BiTSE performance. SI-SNRi decreases as the target speaker moves further away from the front. This is due to the head occlusion effect and the smaller binaural differences in the frontal direction. However, the curves also show that the head occlusion effect does not exhibit monotonic changes. Figure 5 The “Proposed” in the text refers to the voice pickup method of the embodiment of the present application.

[0153] Figure 6 A block diagram of a voice pickup device according to an embodiment of the present application is shown.

[0154] like Figure 6 As shown, the direction-guided voice pickup device 600 includes a transformation module 610 , a first encoding module 620 , a second encoding module 630 , a feature extraction module 640 , and a voice pickup module 650 .

[0155] The transformation module 610 is configured to perform short-time Fourier transform processing on the original speech signal in response to the target user's voice attention instruction to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user;

[0156] The first encoding module 620 is used to perform Boolean direction encoding processing on any direction information to obtain discrete Boolean features;

[0157] A second encoding module 630 is configured to process the plurality of discrete Boolean features using a feature encoder to obtain a first linear feature in a target dimension;

[0158] A feature extraction module 640 is configured to process binaural frequency domain features and multiple orientation information using a spatiotemporal feature extraction network to obtain target convolution features in a target dimension;

[0159] The voice pickup module 650 is used to process the first linear feature and the target convolution feature using the voice extraction network to obtain a target output voice that highlights the voice of the user being focused.

[0160] According to an embodiment of the present application, after the target user gives the direction of attention, the original voice signal is subjected to short-time Fourier transform processing to obtain binaural frequency domain features. For any azimuth information of the focused user, the azimuth information is subjected to Boolean direction encoding processing to obtain discrete Boolean features. Then, a feature encoder is used to process multiple discrete Boolean features to obtain the first linear feature under the target dimension. At the same time, a spatiotemporal feature extraction network is used to process the binaural frequency domain features and multiple azimuth information to obtain the target convolution feature under the target dimension. Finally, a voice extraction network is used to process the first linear feature and the target convolution feature to obtain the target output voice that highlights the voice of the focused user. Because the voice pickup method of the present application clearly distinguishes the direction from which the voice of the focused user reaches the target user through Boolean direction encoding, the ability to extract the voice of the focused user is enhanced without relying on a specific microphone array configuration, thereby improving the clarity of the voice pickup of the focused user.

[0161] According to an embodiment of the present application, the first encoding module 620 includes:

[0162] The mapping submodule is used to perform vector mapping processing on the orientation information to obtain a multi-dimensional vector;

[0163] The time-frequency processing submodule is used to process the multidimensional vector in sequence along the frequency direction and the time dimension direction to obtain discrete Boolean features.

[0164] According to an embodiment of the present application, the time-frequency processing submodule includes:

[0165] A scaling unit is used to scale the multidimensional vector along the frequency direction to obtain a frequency domain feature matrix;

[0166] The expansion unit is used to expand the frequency domain feature matrix along the time dimension to obtain discrete Boolean features.

[0167] According to an embodiment of the present application, the expansion unit includes:

[0168] The first generating subunit is configured to generate discrete Boolean features based on a plurality of frequency domain feature matrices when the target user always follows the followed user;

[0169] a second generating subunit, configured to generate discrete Boolean features based on the attention times of different target paying users and the frequency domain feature matrices of the target paying users at different attention times when the target paying user switches from paying attention to the paying user to paying attention to at least one switching user, wherein the target paying users include the paying user and the at least one switching user, and different target paying users have different location information;

[0170] The third generating subunit is configured to generate discrete Boolean features according to a frequency domain feature matrix under different position information of the focused user when the position information of the focused user changes.

[0171] According to an embodiment of the present application, the second encoding module 630 includes:

[0172] A convolution unit is used to perform two-dimensional convolution processing on the discrete Boolean features corresponding to any orientation information to obtain initial convolution features;

[0173] The first linear unit is used to process multiple initial convolution features of different orientation information using the first linear layer to obtain a first linear feature.

[0174] According to an embodiment of the present application, the binaural frequency domain features include left ear frequency domain features and right ear frequency domain features.

[0175] According to an embodiment of the present application, the feature extraction module 640 includes:

[0176] A direction processing submodule is used to process multiple direction information based on the cosine direction function and the sine direction function respectively, and generate target cosine distances and target sine distances corresponding to different direction information;

[0177] The dimension splicing submodule is used to perform channel dimension splicing processing on the left ear frequency domain features, the right ear frequency domain features, the target cosine distance and the target sine distance to obtain the splicing features;

[0178] The convolution submodule is used to perform one-dimensional convolution on the spliced ​​features to obtain the target convolution features.

[0179] According to an embodiment of the present application, the direction processing submodule includes:

[0180] A first generating unit is configured to generate, for any one of the cosine direction function and the sine direction function, a time difference of arrival corresponding to different direction information according to the average interaural distance, the direction function, and the direction information of the target user;

[0181] The second generating unit is used to generate a theoretical phase difference according to the arrival time difference and frequency information for any azimuth information;

[0182] The third generating unit is configured to generate a target cosine distance or a target sine distance according to the theoretical phase difference, the interaural phase difference, and the direction function, wherein the interaural phase difference is generated based on binaural frequency domain features.

[0183] According to an embodiment of the present application, the third generating unit includes:

[0184] a fourth generating subunit, configured to generate an initial cosine distance or an initial sine distance based on the theoretical phase difference, the interaural phase difference, and the direction function when the target user is always paying attention to the focused user, wherein the initial cosine distance and the initial sine distance represent the target cosine distance and the target sine distance, respectively;

[0185] a fifth generating subunit, configured to generate a target cosine distance or a target sine distance based on initial cosine distances or initial sine distances with target focusing users at different focusing times when the target user switches from focusing on the focused user to focusing on at least one switching user, wherein the target focusing users include the focused user and the at least one switching user, and different target focusing users have different location information;

[0186] The sixth generating subunit is configured to generate a target cosine distance or a target sine distance according to an initial cosine distance or an initial sine distance with different azimuth information from the followed user when the azimuth information of the followed user changes.

[0187] According to an embodiment of the present application, the voice pickup module 650 includes:

[0188] a spatiotemporal fusion unit, configured to process the first linear feature and the target convolution feature based on a Hadamard product to obtain a spatiotemporal fusion feature;

[0189] a deep learning unit, configured to process spatiotemporal fusion features using a target learning network to obtain target recognition features, wherein the target learning network includes a plurality of narrowband converters connected in sequence;

[0190] The second linear unit is used to perform linear processing on the target recognition features to obtain a frequency domain signal;

[0191] The inverse transform unit is used to perform inverse transform processing on the frequency domain signal to obtain the target output speech of the time domain type.

[0192] According to the embodiments of the present application, any number of modules, submodules, units, and subunits, or at least part of the functions of any number of them, can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, which can perform the corresponding functions when the computer program module is executed.

[0193] For example, any number of the transformation module 610, the first encoding module 620, the second encoding module 630, the feature extraction module 640, and the voice pickup module 650 can be combined into a single module / unit / sub-unit, or any one of these modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in a single module / unit / sub-unit. According to an embodiment of the present application, at least one of the transformation module 610, the first encoding module 620, the second encoding module 630, the feature extraction module 640, and the voice pickup module 650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or can be implemented in any one of software, hardware, and firmware, or any suitable combination of any of these. Alternatively, at least one of the transformation module 610, the first encoding module 620, the second encoding module 630, the feature extraction module 640, and the voice pickup module 650 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0194] It should be noted that the voice pickup device part in the embodiment of the present application corresponds to the voice pickup method part in the embodiment of the present application. The description of the voice pickup device part specifically refers to the voice pickup method part and will not be repeated here.

[0195] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0196] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0197] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0198] According to an embodiment of the present application, electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to bus 704. Electronic device 700 may also include one or more of the following components connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or modem. Communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 710 as needed, so that computer programs read from the removable media can be installed into storage section 708 as needed.

[0199] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0200] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0201] According to embodiments of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0202] For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than the ROM 702 and RAM 703 .

[0203] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0204] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present application are performed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0205] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0206] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0207] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The present application does not depart from the scope of the present application, and those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A method for voice pickup with azimuth guidance, characterized in that: include: In response to the target user's voice attention instruction, performing short-time Fourier transform processing on the original voice signal to obtain binaural frequency domain features, wherein the binaural frequency domain features include different orientation information of the focused user; For any of the azimuth information, performing Boolean direction encoding processing on the azimuth information to obtain discrete Boolean features; Processing the plurality of discrete Boolean features using a feature encoder to obtain a first linear feature in a target dimension; Processing the binaural frequency domain features and the plurality of position information using a spatiotemporal feature extraction network to obtain target convolution features under the target dimension; The first linear feature and the target convolution feature are processed using a speech extraction network to obtain a target output speech that highlights the voice of the user being focused.

2. The method according to claim 1, characterized in that The orientation information is subjected to Boolean direction encoding processing to obtain discrete Boolean features, including: Performing vector mapping processing on the orientation information to obtain a multi-dimensional vector; The multidimensional vector is processed in sequence along the frequency direction and the time dimension direction to obtain the discrete Boolean feature.

3. The method according to claim 2, characterized in that The multidimensional vector is processed in sequence along the frequency direction and the time dimension direction to obtain the discrete Boolean feature, including: Scaling the multidimensional vector along the frequency direction to obtain a frequency domain feature matrix; The frequency domain feature matrix is ​​expanded along the time dimension to obtain the discrete Boolean feature.

4. The method according to claim 3, characterized in that The frequency domain feature matrix is ​​expanded along the time dimension to obtain the discrete Boolean feature, including: In a case where the target user always follows the followed user, generating the discrete Boolean feature according to a plurality of the frequency domain feature matrices; In a case where the target user switches from paying attention to the focused user to paying attention to at least one switching user, generating the discrete Boolean feature based on the attention times of different target focused users and the frequency domain feature matrices of the target focused users at different attention times, wherein the target focused users include the focused user and the at least one switching user, and different target focused users have different location information; When the position information of the focused user changes, the discrete Boolean feature is generated according to a frequency domain feature matrix under position information different from that of the focused user.

5. The method according to claim 1, wherein The plurality of discrete Boolean features are processed using a feature encoder to obtain a first linear feature in a target dimension, including: For any of the position information, performing two-dimensional convolution processing on the discrete Boolean features corresponding to the position information to obtain initial convolution features; A first linear layer is used to process the multiple initial convolution features of different orientation information to obtain the first linear feature.

6. The method according to claim 1 or 5, characterized in that The binaural frequency domain features include left ear frequency domain features and right ear frequency domain features; The method of processing the binaural frequency domain features and the plurality of the orientation information using a spatiotemporal feature extraction network to obtain target convolution features under the target dimension includes: Processing the plurality of azimuth information based on the cosine direction function and the sine direction function respectively to generate target cosine distances and target sine distances corresponding to different azimuth information; Performing channel-dimensional splicing processing on the left ear frequency domain feature, the right ear frequency domain feature, the target cosine distance, and the target sine distance to obtain a splicing feature; Perform one-dimensional convolution processing on the splicing features to obtain the target convolution features.

7. The method according to claim 6, characterized in that The plurality of azimuth information are processed based on the cosine direction function and the sine direction function respectively to generate target cosine distances and target sine distances corresponding to different azimuth information, including: For any one of the cosine direction function and the sine direction function, generating arrival time differences corresponding to different direction information according to the average interaural distance of the target user, the direction function, and the direction information; For any of the azimuth information, generating a theoretical phase difference according to the arrival time difference and frequency information; The target cosine distance or the target sine distance is generated according to the theoretical phase difference, the interaural phase difference, and the direction function, wherein the interaural phase difference is generated based on the binaural frequency domain feature.

8. The method according to claim 7, characterized in that Generating the target cosine distance or the target sine distance according to the theoretical phase difference, the interaural phase difference, and the direction function, comprising: In a case where the target user always pays attention to the focused user, generating an initial cosine distance or an initial sine distance according to the theoretical phase difference, the interaural phase difference, and the direction function, wherein the initial cosine distance and the initial sine distance represent the target cosine distance and the target sine distance, respectively; When the target user switches from following the followed user to following at least one switching user, generating the target cosine distance or the target sine distance based on the initial cosine distance or initial sine distance with the target following users at different following times, wherein the target following users include the followed user and the at least one switching user, and different target following users have different location information; When the position information of the focused user changes, the target cosine distance or the target sine distance is generated according to the initial cosine distance or the initial sine distance under position information different from that of the focused user.

9. The method according to claim 1, characterized in that Processing the first linear feature and the target convolutional feature using a speech extraction network to obtain a target output speech that highlights the focused user includes: Processing the first linear feature and the target convolution feature based on the Hadamard product to obtain a spatiotemporal fusion feature; Processing the spatiotemporal fusion features using a target learning network to obtain target recognition features, wherein the target learning network includes a plurality of narrowband converters connected in sequence; Performing linear processing on the target recognition feature to obtain a frequency domain signal; Perform inverse transform processing on the frequency domain signal to obtain the target output speech of the time domain type.

10. A voice pickup device for azimuth guidance, characterized in that: include: a transform module, configured to perform short-time Fourier transform processing on the original voice signal in response to the target user's voice attention instruction to obtain binaural frequency domain features, wherein the binaural frequency domain features include different position information of the focused user; A first encoding module is configured to perform Boolean direction encoding processing on any of the position information to obtain a discrete Boolean feature; A second encoding module is used to process the plurality of discrete Boolean features using a feature encoder to obtain a first linear feature under a target dimension; A feature extraction module, configured to process the binaural frequency domain features and the plurality of position information using a spatiotemporal feature extraction network to obtain a target convolution feature under the target dimension; The voice pickup module is used to process the first linear feature and the target convolution feature using a voice extraction network to obtain a target output voice that highlights the voice of the user being focused.

Images